LFD Book Forum Hoeffding's inequality with M
 User Name Remember Me? Password
 Register FAQ Calendar Mark Forums Read

 Thread Tools Display Modes
#1
07-12-2012, 09:12 PM
 ESRogs Member Join Date: Jul 2012 Posts: 12
Hoeffding's inequality with M

Hi,

I'm wondering why in Lecture 2 -- when we're trying to modify the right hand side of Hoeffding's inequality to take into account drawing from multiple bins (see page 16 of the slides) -- we sum all the probabilities of samples being non-representative, rather than taking 1 - the product of all the probabilities of each sample being representative.

In other words, (calling the right hand side of the original inequality p for simplicity) why doesn't the right hand side of the modified form become 1-(1-p)^M instead of M*p.

Wouldn't that be a still correct but tighter bound?

Thanks!
#2
07-12-2012, 10:17 PM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,478
Re: Hoeffding's inequality with M

Quote:
 Originally Posted by ESRogs Hi, I'm wondering why in Lecture 2 -- when we're trying to modify the right hand side of Hoeffding's inequality to take into account drawing from multiple bins (see page 16 of the slides) -- we sum all the probabilities of samples being non-representative, rather than taking 1 - the product of all the probabilities of each sample being representative. In other words, (calling the right hand side of the original inequality p for simplicity) why doesn't the right hand side of the modified form become 1-(1-p)^M instead of M*p. Wouldn't that be a still correct but tighter bound?
Nice thought. Multiplying would necessitate an independence assumption. While such independence can be easily satisfied in a bin experiment by making sure that different samples are picked independently of each other, the learning counterpart will not follow suit because the different samples are really the same sample (training set) evaluated with different hypotheses.
__________________
Where everyone thinks alike, no one thinks very much
#3
07-12-2012, 11:35 PM
 ESRogs Member Join Date: Jul 2012 Posts: 12
Re: Hoeffding's inequality with M

Oh, I see, that makes sense. The union bound just feels very loose to me, so I was hoping for a tighter one. But it sounds like maybe that's the best we can do without any additional assumptions about the hypothesis set -- is that correct?

Also looking forward to finding out how infinite hypothesis sets are dealt with in future lectures...
#4
07-12-2012, 11:55 PM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,478
Re: Hoeffding's inequality with M

Quote:
 Originally Posted by ESRogs The union bound just feels very loose to me, so I was hoping for a tighter one. But it sounds like maybe that's the best we can do without any additional assumptions about the hypothesis set -- is that correct?
Correct. The theory will use a simple property of the hypthesis set to get a much better bound. Stay tuned!
__________________
Where everyone thinks alike, no one thinks very much

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 08:40 AM.

 Contact Us - LFD Book - Top