LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   General Discussion of Machine Learning (http://book.caltech.edu/bookforum/forumdisplay.php?f=105)
-   -   Normal equation in linear regression (http://book.caltech.edu/bookforum/showthread.php?t=917)

lorddoskias 07-25-2012 02:59 PM

Normal equation in linear regression
 
From coursera's ML course I've known that the normal equation is calculated as follows:

pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y;

Can anyone explain why this is the case?

yaser 07-25-2012 03:12 PM

Re: Normal equation in linear regression
 
Quote:

Originally Posted by lorddoskias (Post 3680)
pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y;

Can anyone explain why this is the case?

pinv is already the pseudo inverse function. When applied to an invertible matrix (in this case X'*X), it returns the regular inverse.

Daniel 09-19-2012 12:06 AM

Re: Normal equation in linear regression
 
Dr. Ng derived the Normal Equation in class, see Lecture 4-6 and he also cautioned about the case where X'*X is non-invertible which meant that there were redundant features (linearly dependent) or too many features (m <= n).

Daniel

magdon 09-20-2012 04:24 AM

Re: Normal equation in linear regression
 
Just a small correction on notation. The normal equations for linear regression are

X^TX w = X^T y.

The solution to the normal equations (for w) is given by the formula that you mention, and indeed the two solutions are equivalent. A proof of this fact is using the singular value decomposition:

X=U\Sigma V^T and X^\dagger=V\Sigma^\dagger U^T

where U^TU=I and V^TV=I. So,

(X^TX)^\dagger X^T=(V\Sigma U^TU\Sigma V^T)^\dagger V\Sigma U^T=V(\Sigma^2)^\dagger V^T V\Sigma U^T=V\Sigma^\dagger U^T=X^\dagger

Quote:

Originally Posted by lorddoskias (Post 3680)
From coursera's ML course I've known that the normal equation is calculated as follows:

pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y;

Can anyone explain why this is the case?


broken857 09-25-2012 01:25 AM

Re: Normal equation in linear regression
 
Thanks you all for useful input. We have some reading to do... :-)


All times are GMT -7. The time now is 04:48 AM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.