LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   The Final (http://book.caltech.edu/bookforum/forumdisplay.php?f=138)
-   -   on the right track? (http://book.caltech.edu/bookforum/showthread.php?t=4104)

Sendai 03-14-2013 06:18 PM

on the right track?
 
Since probably most of us are writing our own regular RBF implementations by hand, I thought it would be helpful to compare the results of a couple simple test cases to make sure our implmentations are correct.

data set = (0, 0) (0, 1) (1, 0) (1, 1)
labels = 1 -1 -1 1
centers = (0, 0.2) (1, 0.7)

Case 1:
\gamma=1
E_{in} = 0
weights = (-7.848, 7.397, 7.823)
(first weight is the bias)

Case 2:
\gamma=100
E_{in} = 0
weights = (-1.0, 109.196, 16206.168)

hemphill 03-14-2013 09:44 PM

Re: on the right track?
 
I concur with those results.

jdreaver 03-14-2013 11:00 PM

Re: on the right track?
 
I agree with those results as well.

ripande 03-14-2013 11:29 PM

Re: on the right track?
 
My results concur with these figures too

kirill 03-15-2013 05:42 PM

Re: on the right track?
 
Same results here.

boulis 03-15-2013 08:00 PM

Re: on the right track?
 
Quote:

Originally Posted by Sendai (Post 9922)
Since probably most of us are writing our own regular RBF implementations by hand, I thought it would be helpful to compare the results of a couple simple test cases to make sure our implmentations are correct.
data set = (0, 0) (0, 1) (1, 0) (1, 1)
labels = 1 -1 -1 1
centers = (0, 0.2) (1, 0.7)

I assume you mean that the centres are given as such and they are not computed by Lloyd's algorithm.
I believe that Lloyd's algorithm would produce unstable results with this configuration of points and number of centers.
Indeed, these are the 3 different cases that I get from running my algorithm (the results depend on the random starting point of course):

centres = [[ 0., 0.], [ 0.66666667, 0.66666667]]
clusters = [[[0, 0]], [[0, 1], [1, 0], [1, 1]]]

centres = [[ 0.66666667, 0.33333333], [ 0., 1.]]
clusters = [[[0, 0], [1, 0], [1, 1]], [[0, 1]]]

centres = [[ 1. , 0.5], [ 0. , 0.5]]
clusters = [[[1, 0], [1, 1]], [[0, 0], [0, 1]]]

These seem pretty reasonable to me. Do other people get the same results?

ripande 03-16-2013 06:23 PM

Re: on the right track?
 
I have run my algo on the last dataset that you have provided and I get the same results

heer2351 03-16-2013 06:28 PM

Re: on the right track?
 
I get the same weights. But do not understand how you get to an Ein of 0 with these points and clusters.

I would say point 1 and 2 belong to center 1, these points have opposite labels. Same for point 3 and 4 which belong to center 2. So how can Ein be zero?

boulis 03-17-2013 02:16 AM

Re: on the right track?
 
Quote:

Originally Posted by heer2351 (Post 9959)
I get the same weights. But do not understand how you get to an Ein of 0 with these points and clusters.

I would say point 1 and 2 belong to center 1, these points have opposite labels. Same for point 3 and 4 which belong to center 2. So how can Ein be zero?

I get the same weights too if I take the centres given as input. And I also get Ein = 0

It is not strange to have Ein = 0, I am not sure why you are confused about it. You are correct that the clusters are as you name them, but the sum of the weighted RBFs is such that we achieve the right sign in the right place (and in fact almost the right value, not just the sign). The key in this case I believe is the bias (W0 or b). This is negative so it gives the field a negative start(sign). Then the two RBFs work to make it positive. The points that are close to the centres are getting affected most and they become positive. The other two are staying negative.

Can someone please revisit my comment earlier? (That these are not the centres when Lloyd's algorithm is applied.)

One more question: Q14 asks the % of times that we get non-separable data by the RBF kernel (i.e. SVM hard margin with RBF kernel). In about 1000 runs that I have tried I never encountered non-separable data. Is this normal? I am using libsvm so it's harder to make a mistake about it... but who knows. I have checked that it identifies non separable data correctly (the 4 point example given in this thread is one such case).

heer2351 03-17-2013 10:33 AM

Re: on the right track?
 
Thanks for your reply, reviewed the lecture again and spotted my thinking error.

I changed my code and it now gives Ein=0 for the examples. I still have doubts however whether my code is correct because when I use Lloyds with these examples my Ein is never zero. As a matter of fact my Ein is rather large all the time; I am trying to find the flaw in my code.

What Ein values do you get in these examples when using Lloyds?

I agree that the centers are not calculated according to Lloyds and also agree with your centers. However I think that there are more centers than you have, for example:

Code:

[1  , 1]  [1/3, 1/3]
[1/2, 1]  [1/2, 0]
[1  , 0]  [1/3, 2/3]

I have the same experience with LibSVM Ein is zero all the time. I attribute this to the fact that this is hard margin SVM and only the number of support vectors goes up. We are basically overfitting which shows in the Eout.

Sendai 03-17-2013 10:45 AM

Re: on the right track?
 
Quote:

Originally Posted by boulis (Post 9942)
I assume you mean that the centres are given as such and they are not computed by Lloyd's algorithm.

Correct.

Quote:

centres = [[ 0., 0.], [ 0.66666667, 0.66666667]]
clusters = [[[0, 0]], [[0, 1], [1, 0], [1, 1]]]

centres = [[ 0.66666667, 0.33333333], [ 0., 1.]]
clusters = [[[0, 0], [1, 0], [1, 1]], [[0, 1]]]

centres = [[ 1. , 0.5], [ 0. , 0.5]]
clusters = [[[1, 0], [1, 1]], [[0, 0], [0, 1]]]
I get these too, plus the ones with x1 and x2 reversed.

heer2351 03-17-2013 10:48 AM

Re: on the right track?
 
@Sendai

What values for Ein do you get when running your algorithm with Lloyds centers?

I think I am still somewhere off the track :(

Sendai 03-17-2013 11:03 AM

Re: on the right track?
 
centres = [[ 0., 0.], [ 0.66666667, 0.66666667]]
Ein = 0

centres = [[ 0.66666667, 0.33333333], [ 0., 1.]]
Ein = 0

centres = [[ 1. , 0.5], [ 0. , 0.5]]
Ein = 0.5
(The weight are all zeros -- RBF can't break the symmetry and fails.)

Anne Paulson 03-17-2013 04:38 PM

Re: on the right track?
 
I had a difficult bug to find here, in R. Beware, other R users. I ran the first example, and it was just fine. I ran the second example, and got nothing like the right answer. What? said me.

I had been using my own linear regression program, since it was conveniently to hand. But I knew it worked. So I tried the builtin R linear regression, which gave the correct answer. I noticed that in my own linear regression, I was calling ginv, the pseudo-inverse function. I called solve, the actual inverse function (why you get the inverse of a square matrix by calling "solve" is beyond me, but I digress). That worked.

Finally I realized that our little example is quite a nasty matrix. The tolerance for ginv was something like 1E-8. Not small enough! Once I lowered the tolerance, everything was dandy.

boulis 03-17-2013 05:29 PM

Re: on the right track?
 
Thanks heer2351 and Sendai. Indeed there are 6 different centre-configurations, the ones you gave complete the picture. You can also see this with paper&pencil.

Quote:

Originally Posted by heer2351 (Post 9974)
@Sendai

What values for Ein do you get when running your algorithm with Lloyds centers?
I think I am still somewhere off the track :(

With this 4-point example the only options for Ein are: 0, 0.25, 0.5
Anything else you get, you know your code has a bug.

I have not tried it with my code, but Sendai's results seem correct. So you should get Ein =0 for the centres that have 1/3 or 2/3 in them (when the clusters are 3-1 points) and Ein = 0.5 when the centres have 1/2 in them (the case were the clusters are 2-2points)

I also think that it might be impossible to get Ein= 0.25 for any centre configuration, providing that the weights are chosen optimally (i.e., using the pseudo inverse method). So you either make no mistakes, or 2 points are misclassified.

melipone 03-18-2013 07:21 AM

Re: on the right track?
 
Geez, I don't get the same weights! I do get Ein=0 though. Could you post your phi matrix so that I can locate my error? thanks.

Quote:

Originally Posted by Sendai (Post 9922)
Since probably most of us are writing our own regular RBF implementations by hand, I thought it would be helpful to compare the results of a couple simple test cases to make sure our implmentations are correct.

data set = (0, 0) (0, 1) (1, 0) (1, 1)
labels = 1 -1 -1 1
centers = (0, 0.2) (1, 0.7)

Case 1:
\gamma=1
E_{in} = 0
weights = (-7.848, 7.397, 7.823)
(first weight is the bias)

Case 2:
\gamma=100
E_{in} = 0
weights = (-1.0, 109.196, 16206.168)


Sendai 03-18-2013 10:04 AM

Re: on the right track?
 
Quote:

Originally Posted by melipone (Post 9996)
geez, i don't get the same weights! I do get ein=0 though. Could you post your phi matrix so that i can locate my error? Thanks.

Here's the phi matrix I get for case 1:

Code:

[[ 1.          0.96078944  0.22537266]
 [ 1.          0.35345468  0.61262639]
 [ 1.          0.52729242  0.33621649]
 [ 1.          0.19398004  0.91393119]]


heer2351 03-18-2013 05:53 PM

Re: on the right track?
 
Sendai thanks for starting this thread, it enabled me to find a flaw in my code and even better fix it :)

All answers I found with my fixed code were correct.

Special thanks to boulis for his first response, I was overthinking the problem.

melipone 03-18-2013 07:22 PM

Re: on the right track?
 
Ah ha, I see how you got that but I think that you need to take the square root after you add up the distance of each data point to the cluster centers. You did not take the square root. I think you have to IMHO.

Quote:

Originally Posted by Sendai (Post 9997)
Here's the phi matrix I get for case 1:

Code:

[[ 1.          0.96078944  0.22537266]
 [ 1.          0.35345468  0.61262639]
 [ 1.          0.52729242  0.33621649]
 [ 1.          0.19398004  0.91393119]]



boulis 03-18-2013 09:41 PM

Re: on the right track?
 
Quote:

Originally Posted by melipone (Post 10008)
Ah ha, I see how you got that but I think that you need to take the square root after you add up the distance of each data point to the cluster centers. You did not take the square root. I think you have to IMHO.

If you see the formula, you'll notice that the norm is getting squared. So no need to take the square root in the first place.

melipone 03-19-2013 06:32 AM

Re: on the right track?
 
The answers indicate to me that you should take the square root. At least, I had the answers right before I changed them because of that. :(

tonytony 03-19-2013 07:22 AM

Re: on the right track?
 
Hi, could please someone post the right PHI matrix?
in order to know if we neeed to use the norm or the norm squared?
thanks!


All times are GMT -7. The time now is 06:40 AM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.