LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 3

Reply
 
Thread Tools Display Modes
  #1  
Old 04-21-2013, 06:31 AM
skwong skwong is offline
Member
 
Join Date: Apr 2013
Location: Hong Kong
Posts: 13
Default slide #4 lecture 6 'Recursive Bound on B(N,k)'

I have difficulty in understanding this. I tried to think about it by using
numerical example, say with the Perception, k = 4; and using N = 4 (0 = -1,
1 = +1).

Say, for alpha, I ignore x4 (don't care terms), then I can list all 8
combination of x1 to x3; when x4 is 0, then only 3 case works; and the
opposite is same. Therefore, I can come up with following:

x1 x2 x3 | x4

0 0 0 1
1 0 0 0
0 1 0 0
1 1 0 1
0 0 1 0
1 0 1 1
0 1 1 1
1 1 1 0

0 0 0 0
0 1 1 0
#1 0 1 0 (that should be left out)
1 1 0 0

1 1 1 1
1 0 0 1
#0 1 0 1 (that should be left out)
0 0 1 1

Then, I found that I mis-understood. Because:

1) The total is 14 (which matches with what was found in previous lecture),
but does not match with the B(N, k) calculation result of 15.

2) According to the numerical table in slide 8/18, the beta is 7, alpha
should be 1. I try to list them out like the above, but cannot know
which 7 combination to pick for beta rows, because I think I need to
kick out the 1010 and 0101 pattern. But then the 2 sections of beta
in the table will not be same for the x1 to x3.

Therefore, I actually do not understand it. Can you please enlighten me ?

Thanks in advance.

SK
Reply With Quote
  #2  
Old 04-21-2013, 06:58 AM
Elroch Elroch is offline
Invited Guest
 
Join Date: Mar 2013
Posts: 143
Default Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

B(N, k) is the most dichotomies there can be. When N >= k, the actual number can take a range of values between 2^{k-1} and B(N, k), where k=d_{VC}+1 (another restriction on this variation is that it is an non-decreasing function of N).
Reply With Quote
  #3  
Old 04-21-2013, 11:48 AM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:
Originally Posted by skwong View Post
1) The total is 14 (which matches with what was found in previous lecture),
but does not match with the B(N, k) calculation result of 15.

2) According to the numerical table in slide 8/18, the beta is 7, alpha
should be 1. I try to list them out like the above, but cannot know
which 7 combination to pick for beta rows, because I think I need to
kick out the 1010 and 0101 pattern. But then the 2 sections of beta
in the table will not be same for the x1 to x3.
1) The total is indeed 14 for the matrix you constructed, and for the perceprton case. The maximum possible is 15, though (take all posible patterns of 4 bits except the all 0's, and you have a break point of 4). Since B(N,k) is an upper bound, there is no problem here.

2) In the matrix that excludes the all 0's, the set S_1 contains only the pattern 0001, so \alpha=1, while the set S_2 contains the remaining 14 patterns (excluding all 0's) so \beta=7.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #4  
Old 04-21-2013, 12:25 PM
Elroch Elroch is offline
Invited Guest
 
Join Date: Mar 2013
Posts: 143
Default Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

I should perhaps explain the thinking behind my last post on the general bounds on the variation of growth functions (so someone can check it ). The minimum number of dichotomies on a set of N points where N\geq k is given by the fact that we know some set of k-1 points is shattered, and we can extend such a shattering in a minimal way to the whole set of points in a way that excludes all other points. This makes the growth function constant 2^{k-1} for all N>=k.

I believe all the possible values of the growth function between this minimum and the bound B(N, k) can occur for some carefully chosen hypothesis set, but haven't actually proved it. Is this so?
Reply With Quote
  #5  
Old 04-23-2013, 03:39 AM
skwong skwong is offline
Member
 
Join Date: Apr 2013
Location: Hong Kong
Posts: 13
Default Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:
Originally Posted by yaser View Post
1) The total is indeed 14 for the matrix you constructed, and for the perceprton case. The maximum possible is 15, though (take all posible patterns of 4 bits except the all 0's, and you have a break point of 4). Since B(N,k) is an upper bound, there is no problem here.

2) In the matrix that excludes the all 0's, the set S_1 contains only the pattern 0001, so \alpha=1, while the set S_2 contains the remaining 14 patterns (excluding all 0's) so \beta=7.
Thanks for the reply. For the 1), I suspected that 14 is for perceptron and
15 is upper bound and do no harm. That confirmed my thought.

For the original subject, I did not understand. After repeatedly watching the
video and thinking, I eventually understood the motive and reasoning behind.

But then my question is, would it be possible to think it the other way:

First group the rows with k-2 column, for the N = 4, k = 4 case (and now
name this as half of S_2 with \beta=3 rows:

0 1 1 0
1 0 1 0
1 1 0 0

Then, make the complementary

1 0 0 1
0 1 0 1
0 0 1 1

And let the remaining as \alpha=8:

0 0 0 x
0 0 1 0
0 1 0 0
0 1 1 1
1 0 0 0
1 0 1 1
1 1 0 1
1 1 1 x

And the calculation will be
\sum\limits_{i=0}^{k-1} {N - 1 \choose i} + \sum\limits_{i=0}^{k-2} {N - 2 \choose i}

Then, obviously, when k = N, the B(N, k) will not be evaluated to
2^k - 1

Am I missing something ?
Reply With Quote
  #6  
Old 04-23-2013, 09:49 AM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:
Originally Posted by skwong View Post
for the N = 4, k = 4 case (and now
name this as half of S_2 with \beta=3 rows:

0 1 1 0
1 0 1 0
1 1 0 0

Then, make the complementary

1 0 0 1
0 1 0 1
0 0 1 1

And let the remaining as \alpha=8:

0 0 0 x
0 0 1 0
0 1 0 0
0 1 1 1
1 0 0 0
1 0 1 1
1 1 0 1
1 1 1 x
I did not understand this construction. The two parts of S_2 (the first two matrices with 3 rows) should be complements to each other only in the last column, but identical in the first 3. Also, when you focus on the first 3 columns, there should be no common rows between S_2 and S_1 (the latter being the matrix with 8 rows), and in your construction there are common rows.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 04:27 AM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.