LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Homework 2 (http://book.caltech.edu/bookforum/forumdisplay.php?f=131)
-   -   Multiplication matrix X and its transpose (http://book.caltech.edu/bookforum/showthread.php?t=4498)

tarquilu 09-15-2014 04:33 PM

Multiplication matrix X and its transpose
 
Hello,

I'm trying to compute the pseudo-inverse of matrix X with dimensions [d+1 x N], where N is the number of data points. Lecture 3, slide 16 shows that the dimensions "shrink" when multiplying (X^T)X from [d+1 x N][N x d+1] to [d+1 x d+1].

Although LFD on page 86 states that "In practice, (X^T)T is invertible in most of the cases since N is often much bigger than d+1, so there will likely be d+1 linearly independent vectors xn," I don't know if this statement is meant to explain the reason for the simplification to [d+1 x d+1] dimensions. Moreover, when creating X and computing (X^T)T, I obtain a matrix with [N x N] dimensions, not [d+1 x d+1] dimensions.

Could some please clarify this for me? Thank you.

yaser 09-16-2014 01:02 AM

Re: Multiplication matrix X and its transpose
 
Quote:

Originally Posted by tarquilu (Post 11725)
Hello,

I'm trying to compute the pseudo-inverse of matrix X with dimensions [d+1 x N], where N is the number of data points. Lecture 3, slide 16 shows that the dimensions "shrink" when multiplying (X^T)X from [d+1 x N][N x d+1] to [d+1 x d+1].

Although LFD on page 86 states that "In practice, (X^T)T is invertible in most of the cases since N is often much bigger than d+1, so there will likely be d+1 linearly independent vectors xn," I don't know if this statement is meant to explain the reason for the simplification to [d+1 x d+1] dimensions. Moreover, when creating X and computing (X^T)T, I obtain a matrix with [N x N] dimensions, not [d+1 x d+1] dimensions.

Could some please clarify this for me? Thank you.

1. X is in fact of dimensions N x d+1. The transpose is d+1 x N.

2. The remark about independent vectors is only meant to argue that the d+1 x d+1 matrix has full rank (d+1) since in principle it could be singular, with rank less than d+1 hence not invertible.

3. I am not clear on what (X^T)T is.

tarquilu 09-21-2014 11:14 AM

Re: Multiplication matrix X and its transpose
 
Thank you professor. Sorry, I meant to write (X^T)X to symbolize the transpose of matrix X multiplied by X.

I now know why my matrix was not "shrinking." I was multiplying X(X^T) instead of (X^T)X so that is why I was not obtaining a [d+1 x d+1] dimensional matrix.

However, the g hypotheses I'm obtaining with w = (X^dagger)y are usually not close to target functions f... random, really. I'm obtaining an average Ein of approximately 0.5. Sometimes my code gives a g with an extremely large or small y-intercept value (e.g., 3.60288*10^16 - 0.5x). Any pointers would be appreciated. Thank you.

yaser 09-24-2014 09:15 PM

Re: Multiplication matrix X and its transpose
 
Just want to point out that a regular session of the course is starting this week at edX:

http://www.edx.org/course/caltechx/c...ning-data-2516

I am attending to the discussion forum over there, so I'll check here less frequently during that session.

tarquilu 10-14-2014 04:47 PM

Re: Multiplication matrix X and its transpose
 
I made a simple program coding error which was leading to the aberrant weights and Ein. Once I fixed the coding error, I obtained the correct weights and Ein.


All times are GMT -7. The time now is 07:43 PM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.