Multiplication matrix X and its transpose

[tarquilu]

Hello,

I'm trying to compute the pseudo-inverse of matrix X with dimensions [d+1 x N], where N is the number of data points. Lecture 3, slide 16 shows that the dimensions "shrink" when multiplying (X^T)X from [d+1 x N][N x d+1] to [d+1 x d+1].

Although LFD on page 86 states that "In practice, (X^T)T is invertible in most of the cases since N is often much bigger than d+1, so there will likely be d+1 linearly independent vectors xn," I don't know if this statement is meant to explain the reason for the simplification to [d+1 x d+1] dimensions. Moreover, when creating X and computing (X^T)T, I obtain a matrix with [N x N] dimensions, not [d+1 x d+1] dimensions.

Could some please clarify this for me? Thank you.

[yaser]

Originally Posted by tarquilu

Hello,

I'm trying to compute the pseudo-inverse of matrix X with dimensions [d+1 x N], where N is the number of data points. Lecture 3, slide 16 shows that the dimensions "shrink" when multiplying (X^T)X from [d+1 x N][N x d+1] to [d+1 x d+1].

Although LFD on page 86 states that "In practice, (X^T)T is invertible in most of the cases since N is often much bigger than d+1, so there will likely be d+1 linearly independent vectors xn," I don't know if this statement is meant to explain the reason for the simplification to [d+1 x d+1] dimensions. Moreover, when creating X and computing (X^T)T, I obtain a matrix with [N x N] dimensions, not [d+1 x d+1] dimensions.

Could some please clarify this for me? Thank you.

1. X is in fact of dimensions N x d+1. The transpose is d+1 x N.

2. The remark about independent vectors is only meant to argue that the d+1 x d+1 matrix has full rank (d+1) since in principle it could be singular, with rank less than d+1 hence not invertible.

3. I am not clear on what (X^T)T is.

[tarquilu]

Thank you professor. Sorry, I meant to write (X^T)X to symbolize the transpose of matrix X multiplied by X.

I now know why my matrix was not "shrinking." I was multiplying X(X^T) instead of (X^T)X so that is why I was not obtaining a [d+1 x d+1] dimensional matrix.

However, the g hypotheses I'm obtaining with w = (X^dagger)y are usually not close to target functions f... random, really. I'm obtaining an average Ein of approximately 0.5. Sometimes my code gives a g with an extremely large or small y-intercept value (e.g., 3.60288*10^16 - 0.5x). Any pointers would be appreciated. Thank you.

[yaser]

Just want to point out that a regular session of the course is starting this week at edX:

http://www.edx.org/course/caltechx/c...ning-data-2516

I am attending to the discussion forum over there, so I'll check here less frequently during that session.

[tarquilu]

I made a simple program coding error which was leading to the aberrant weights and Ein. Once I fixed the coding error, I obtained the correct weights and Ein.