View Single Post
Old 09-20-2012, 04:24 AM
magdon's Avatar
magdon magdon is offline
Join Date: Aug 2009
Location: Troy, NY, USA.
Posts: 597
Default Re: Normal equation in linear regression

Just a small correction on notation. The normal equations for linear regression are

X^TX w = X^T y.

The solution to the normal equations (for w) is given by the formula that you mention, and indeed the two solutions are equivalent. A proof of this fact is using the singular value decomposition:

X=U\Sigma V^T and X^\dagger=V\Sigma^\dagger U^T

where U^TU=I and V^TV=I. So,

(X^TX)^\dagger X^T=(V\Sigma U^TU\Sigma V^T)^\dagger V\Sigma U^T=V(\Sigma^2)^\dagger V^T V\Sigma U^T=V\Sigma^\dagger U^T=X^\dagger

Originally Posted by lorddoskias View Post
From coursera's ML course I've known that the normal equation is calculated as follows:

pinv((X'*X))*X'*Y; (octave code) but apparently this is equivalent to just pinv(X)*Y;

Can anyone explain why this is the case?
Have faith in probability
Reply With Quote