LFD Book Forum A little puzzle

#1
04-18-2013, 06:25 AM
 Elroch Invited Guest Join Date: Mar 2013 Posts: 143
A little puzzle

Pondering the loose relationship between parametrisation of a hypothesis set and VC dimension (or the minimum break point, ) led me to the following example and puzzle.

Let the set of points be the natural numbers

Let elements of the hypothesis set be made up of alternating intervals of the same size, like a 1-dimensional checkerboard with varying scale

What is the VC dimension of this hypothesis set?

[there is also a continuous version on the real line, but all the structure is in this simplified version]
#2
04-18-2013, 04:15 PM
 Michael Reach Senior Member Join Date: Apr 2013 Location: Baltimore, Maryland, USA Posts: 71
Re: A little puzzle

Don't have the rules too clear. Can your alternating segments contain the +1s, _or_ all the -1s? If not, I imagine the VC dimension is 0.

If you allow both, I guess the first pattern you can't achieve is +1, -1, -1
so the VC dimension would be 2.

Or do you allow them shifted as well, then you'd get that one by starting at 2. But I don't think you could do +1, -1, +1, +1.
#3
04-18-2013, 04:44 PM
 Elroch Invited Guest Join Date: Mar 2013 Posts: 143
Re: A little puzzle

Forgive me if it wasn't clear. There is exactly one hypothesis for each positive integer , as described in the first post.

Intuitively, each hypothesis (i.e. permitted subset of )is an alternating sequence of "black" and "white" intervals of equal size starting at 1, and continuing indefinitely. "Black" = +1 = inclusion in the set, and "white" = -1 = exclusion from the set.

ok?

Do remember you have great freedom as to how to choose a set of N points. Use it well.

A good attack might be to try to shatter sets for increasing from upward.
#4
04-18-2013, 06:41 PM
 Michael Reach Senior Member Join Date: Apr 2013 Location: Baltimore, Maryland, USA Posts: 71
Re: A little puzzle

Ah - not the first time today I was tripped up by the rule that we only need to find a single set of xns with the maximum shattering. I keep forgetting and looking for some set of xns which doesn't.

So here's a method that will work for any N, and I don't even need all of your hypotheses; it's enough just to use the set of H's where n=2^k.
H0=+1 for 1,3,5,7,...
H1=+1 for 1,2,5,6,9,10,...
H2=+1 for 1,2,3,4,9,10,11,12,17,18,19,20,...
H3=+1 for 1,2,3,4,5,6,7,8,17,18,19,...

The easy way to see it is to use binary notation.
H0=+1 for any n with the last digit=1
H1=+1 for any n with the second-to-last digit=1
H2=+1 for any n with the third-to-last digit=1 etc.

Now gH(N)=2^N. Here's how we build our set of x1,...,xN that we can shatter.

For N, take the set Z of every combination of N 1s and 0s and use them to build our x1,...,xN. They will be very big numbers indeed, with 2^N digits or so.
The first digit of xi, call it xi1, is the ith digit of the first element of Z. If that is 000000...0 (N 0s), the first digits of all the xis will all be zero as well, so all of the numbers x1,...,xN will start with 0.
If the second element of Z is 000000..01 - might as well keep them in order - the second digit of x1,...,xN will be 0, 0, ..., 0, and 1 respectively. And so forth, for all 2^N digits.

These N very special binary numbers are shattered by H0 through H(2^N-1), as H(i) picks out the ith digit of each of our numbers, and the digits cover every possibility in Z.

Way to go - this is a neat problem.
#5
04-18-2013, 06:43 PM
 Elroch Invited Guest Join Date: Mar 2013 Posts: 143
Re: A little puzzle

Thank you and well done!

Exactly the solution I found.
#6
04-18-2013, 09:07 PM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,477
Re: A little puzzle

Cool. I'll copy the thread into the Create New Homework Problems forum.
__________________
Where everyone thinks alike, no one thinks very much

 Thread Tools Display Modes Hybrid Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 05:31 PM.