LFD Book Forum - slide #4 lecture 6 'Recursive Bound on B(N,k)'

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)

- Homework 3 (http://book.caltech.edu/bookforum/forumdisplay.php?f=132)

- - slide #4 lecture 6 'Recursive Bound on B(N,k)' (http://book.caltech.edu/bookforum/showthread.php?t=4227)

slide #4 lecture 6 'Recursive Bound on B(N,k)'

I have difficulty in understanding this. I tried to think about it by using
numerical example, say with the Perception, k = 4; and using N = 4 (0 = -1,
1 = +1).

Say, for alpha, I ignore x4 (don't care terms), then I can list all 8
combination of x1 to x3; when x4 is 0, then only 3 case works; and the
opposite is same. Therefore, I can come up with following:

x1 x2 x3 | x4

0 0 0 1
1 0 0 0
0 1 0 0
1 1 0 1
0 0 1 0
1 0 1 1
0 1 1 1
1 1 1 0

0 0 0 0
0 1 1 0
#1 0 1 0 (that should be left out)
1 1 0 0

1 1 1 1
1 0 0 1
#0 1 0 1 (that should be left out)
0 0 1 1

Then, I found that I mis-understood. Because:

1) The total is 14 (which matches with what was found in previous lecture),
but does not match with the B(N, k) calculation result of 15.

2) According to the numerical table in slide 8/18, the beta is 7, alpha
should be 1. I try to list them out like the above, but cannot know
which 7 combination to pick for beta rows, because I think I need to
kick out the 1010 and 0101 pattern. But then the 2 sections of beta
in the table will not be same for the x1 to x3.

Therefore, I actually do not understand it. Can you please enlighten me ?

Thanks in advance.

SK

Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

is the most dichotomies there can be. When

, the actual number can take a range of values between $2^{k-1}$ and

, where $k=d_{VC}+1$ (another restriction on this variation is that it is an non-decreasing function of

Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:

Originally Posted by skwong (Post 10525)

1) The total is 14 (which matches with what was found in previous lecture),
but does not match with the B(N, k) calculation result of 15.

2) According to the numerical table in slide 8/18, the beta is 7, alpha
should be 1. I try to list them out like the above, but cannot know
which 7 combination to pick for beta rows, because I think I need to
kick out the 1010 and 0101 pattern. But then the 2 sections of beta
in the table will not be same for the x1 to x3.

1) The total is indeed 14 for the matrix you constructed, and for the perceprton case. The maximum possible is 15, though (take all posible patterns of 4 bits except the all 0's, and you have a break point of 4). Since

is an upper bound, there is no problem here.

2) In the matrix that excludes the all 0's, the set

contains only the pattern 0001, so $\alpha=1$ , while the set

contains the remaining 14 patterns (excluding all 0's) so $\beta=7$ .

Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

I should perhaps explain the thinking behind my last post on the general bounds on the variation of growth functions (so someone can check it :) ). The minimum number of dichotomies on a set of

points where $N\geq k$ is given by the fact that we know some set of

points is shattered, and we can extend such a shattering in a minimal way to the whole set of points in a way that excludes all other points. This makes the growth function constant $2^{k-1}$ for all

.

I believe all the possible values of the growth function between this minimum and the bound B(N, k) can occur for some carefully chosen hypothesis set, but haven't actually proved it. Is this so?

Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:

Originally Posted by yaser (Post 10537)

is an upper bound, there is no problem here.

2) In the matrix that excludes the all 0's, the set

contains only the pattern 0001, so $\alpha=1$ , while the set

contains the remaining 14 patterns (excluding all 0's) so $\beta=7$ .

Thanks for the reply. For the 1), I suspected that 14 is for perceptron and
15 is upper bound and do no harm. That confirmed my thought.

For the original subject, I did not understand. After repeatedly watching the
video and thinking, I eventually understood the motive and reasoning behind.

But then my question is, would it be possible to think it the other way:

First group the rows with k-2 column, for the N = 4, k = 4 case (and now
name this as half of

with $\beta=3$ rows:

0 1 1 0
1 0 1 0
1 1 0 0

Then, make the complementary

1 0 0 1
0 1 0 1
0 0 1 1

And let the remaining as $\alpha=8$ :

0 0 0 x
0 0 1 0
0 1 0 0
0 1 1 1
1 0 0 0
1 0 1 1
1 1 0 1
1 1 1 x

And the calculation will be
$\sum\limits_{i=0}^{k-1} {N - 1 \choose i} + \sum\limits_{i=0}^{k-2} {N - 2 \choose i}$

Then, obviously, when k = N, the B(N, k) will not be evaluated to

Am I missing something ?

Re: slide #4 lecture 6 'Recursive Bound on B(N,k)'

Quote:

Originally Posted by skwong (Post 10554)

for the N = 4, k = 4 case (and now
name this as half of

with $\beta=3$ rows:

0 1 1 0
1 0 1 0
1 1 0 0

Then, make the complementary

1 0 0 1
0 1 0 1
0 0 1 1

And let the remaining as $\alpha=8$ :

0 0 0 x
0 0 1 0
0 1 0 0
0 1 1 1
1 0 0 0
1 0 1 1
1 1 0 1
1 1 1 x

I did not understand this construction. The two parts of

(the first two matrices with 3 rows) should be complements to each other only in the last column, but identical in the first 3. Also, when you focus on the first 3 columns, there should be no common rows between

and

(the latter being the matrix with 8 rows), and in your construction there are common rows.