Normal equation in linear regression

[lorddoskias]

From coursera's ML course I've known that the normal equation is calculated as follows:

pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y;

Can anyone explain why this is the case?

[yaser]

Originally Posted by lorddoskias

pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y;

Can anyone explain why this is the case?

pinv is already the pseudo inverse function. When applied to an invertible matrix (in this case X'*X), it returns the regular inverse.

[Daniel]

Dr. Ng derived the Normal Equation in class, see Lecture 4-6 and he also cautioned about the case where X'*X is non-invertible which meant that there were redundant features (linearly dependent) or too many features (m <= n).

Daniel

[magdon]

Just a small correction on notation. The normal equations for linear regression are



The solution to the normal equations (for ) is given by the formula that you mention, and indeed the two solutions are equivalent. A proof of this fact is using the singular value decomposition:

and

where and . So,

Originally Posted by lorddoskias

From coursera's ML course I've known that the normal equation is calculated as follows:

pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y;

Can anyone explain why this is the case?

[broken857]

Thanks you all for useful input. We have some reading to do... :-)