LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   General Discussion of Machine Learning (http://book.caltech.edu/bookforum/forumdisplay.php?f=105)
-   -   Normal equation in linear regression (http://book.caltech.edu/bookforum/showthread.php?t=917)

 lorddoskias 07-25-2012 01:59 PM

Normal equation in linear regression

From coursera's ML course I've known that the normal equation is calculated as follows:

pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y;

Can anyone explain why this is the case?

 yaser 07-25-2012 02:12 PM

Re: Normal equation in linear regression

Quote:
 Originally Posted by lorddoskias (Post 3680) pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y; Can anyone explain why this is the case?
pinv is already the pseudo inverse function. When applied to an invertible matrix (in this case X'*X), it returns the regular inverse.

 Daniel 09-18-2012 11:06 PM

Re: Normal equation in linear regression

Dr. Ng derived the Normal Equation in class, see Lecture 4-6 and he also cautioned about the case where X'*X is non-invertible which meant that there were redundant features (linearly dependent) or too many features (m <= n).

Daniel

 magdon 09-20-2012 03:24 AM

Re: Normal equation in linear regression

Just a small correction on notation. The normal equations for linear regression are

The solution to the normal equations (for ) is given by the formula that you mention, and indeed the two solutions are equivalent. A proof of this fact is using the singular value decomposition:

and

where and . So,

Quote:
 Originally Posted by lorddoskias (Post 3680) From coursera's ML course I've known that the normal equation is calculated as follows: pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y; Can anyone explain why this is the case?

 broken857 09-25-2012 12:25 AM

Re: Normal equation in linear regression

Thanks you all for useful input. We have some reading to do... :-)

 All times are GMT -7. The time now is 02:57 PM.