LFD Book Forum slide #4 lecture 6 'Recursive Bound on B(N,k)'
 Register FAQ Calendar Mark Forums Read

#1
04-21-2013, 05:31 AM
 skwong Member Join Date: Apr 2013 Location: Hong Kong Posts: 13
slide #4 lecture 6 'Recursive Bound on B(N,k)'

I have difficulty in understanding this. I tried to think about it by using
numerical example, say with the Perception, k = 4; and using N = 4 (0 = -1,
1 = +1).

Say, for alpha, I ignore x4 (don't care terms), then I can list all 8
combination of x1 to x3; when x4 is 0, then only 3 case works; and the
opposite is same. Therefore, I can come up with following:

x1 x2 x3 | x4

0 0 0 1
1 0 0 0
0 1 0 0
1 1 0 1
0 0 1 0
1 0 1 1
0 1 1 1
1 1 1 0

0 0 0 0
0 1 1 0
#1 0 1 0 (that should be left out)
1 1 0 0

1 1 1 1
1 0 0 1
#0 1 0 1 (that should be left out)
0 0 1 1

Then, I found that I mis-understood. Because:

1) The total is 14 (which matches with what was found in previous lecture),
but does not match with the B(N, k) calculation result of 15.

2) According to the numerical table in slide 8/18, the beta is 7, alpha
should be 1. I try to list them out like the above, but cannot know
which 7 combination to pick for beta rows, because I think I need to
kick out the 1010 and 0101 pattern. But then the 2 sections of beta
in the table will not be same for the x1 to x3.

Therefore, I actually do not understand it. Can you please enlighten me ?

SK
#2
04-21-2013, 05:58 AM
 Elroch Invited Guest Join Date: Mar 2013 Posts: 143
Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

is the most dichotomies there can be. When , the actual number can take a range of values between and , where (another restriction on this variation is that it is an non-decreasing function of ).
#3
04-21-2013, 10:48 AM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,478
Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:
 Originally Posted by skwong 1) The total is 14 (which matches with what was found in previous lecture), but does not match with the B(N, k) calculation result of 15. 2) According to the numerical table in slide 8/18, the beta is 7, alpha should be 1. I try to list them out like the above, but cannot know which 7 combination to pick for beta rows, because I think I need to kick out the 1010 and 0101 pattern. But then the 2 sections of beta in the table will not be same for the x1 to x3.
1) The total is indeed 14 for the matrix you constructed, and for the perceprton case. The maximum possible is 15, though (take all posible patterns of 4 bits except the all 0's, and you have a break point of 4). Since is an upper bound, there is no problem here.

2) In the matrix that excludes the all 0's, the set contains only the pattern 0001, so , while the set contains the remaining 14 patterns (excluding all 0's) so .
__________________
Where everyone thinks alike, no one thinks very much
#4
04-21-2013, 11:25 AM
 Elroch Invited Guest Join Date: Mar 2013 Posts: 143
Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

I should perhaps explain the thinking behind my last post on the general bounds on the variation of growth functions (so someone can check it ). The minimum number of dichotomies on a set of points where is given by the fact that we know some set of points is shattered, and we can extend such a shattering in a minimal way to the whole set of points in a way that excludes all other points. This makes the growth function constant for all .

I believe all the possible values of the growth function between this minimum and the bound B(N, k) can occur for some carefully chosen hypothesis set, but haven't actually proved it. Is this so?
#5
04-23-2013, 02:39 AM
 skwong Member Join Date: Apr 2013 Location: Hong Kong Posts: 13
Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:
 Originally Posted by yaser 1) The total is indeed 14 for the matrix you constructed, and for the perceprton case. The maximum possible is 15, though (take all posible patterns of 4 bits except the all 0's, and you have a break point of 4). Since is an upper bound, there is no problem here. 2) In the matrix that excludes the all 0's, the set contains only the pattern 0001, so , while the set contains the remaining 14 patterns (excluding all 0's) so .
Thanks for the reply. For the 1), I suspected that 14 is for perceptron and
15 is upper bound and do no harm. That confirmed my thought.

For the original subject, I did not understand. After repeatedly watching the
video and thinking, I eventually understood the motive and reasoning behind.

But then my question is, would it be possible to think it the other way:

First group the rows with k-2 column, for the N = 4, k = 4 case (and now
name this as half of with rows:

0 1 1 0
1 0 1 0
1 1 0 0

Then, make the complementary

1 0 0 1
0 1 0 1
0 0 1 1

And let the remaining as :

0 0 0 x
0 0 1 0
0 1 0 0
0 1 1 1
1 0 0 0
1 0 1 1
1 1 0 1
1 1 1 x

And the calculation will be

Then, obviously, when k = N, the B(N, k) will not be evaluated to

Am I missing something ?
#6
04-23-2013, 08:49 AM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,478
Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:
 Originally Posted by skwong for the N = 4, k = 4 case (and now name this as half of with rows: 0 1 1 0 1 0 1 0 1 1 0 0 Then, make the complementary 1 0 0 1 0 1 0 1 0 0 1 1 And let the remaining as : 0 0 0 x 0 0 1 0 0 1 0 0 0 1 1 1 1 0 0 0 1 0 1 1 1 1 0 1 1 1 1 x
I did not understand this construction. The two parts of (the first two matrices with 3 rows) should be complements to each other only in the last column, but identical in the first 3. Also, when you focus on the first 3 columns, there should be no common rows between and (the latter being the matrix with 8 rows), and in your construction there are common rows.
__________________
Where everyone thinks alike, no one thinks very much

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 08:23 PM.