LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 1

Reply
 
Thread Tools Display Modes
  #1  
Old 07-12-2012, 09:12 PM
ESRogs ESRogs is offline
Member
 
Join Date: Jul 2012
Posts: 12
Default Hoeffding's inequality with M

Hi,

I'm wondering why in Lecture 2 -- when we're trying to modify the right hand side of Hoeffding's inequality to take into account drawing from multiple bins (see page 16 of the slides) -- we sum all the probabilities of samples being non-representative, rather than taking 1 - the product of all the probabilities of each sample being representative.

In other words, (calling the right hand side of the original inequality p for simplicity) why doesn't the right hand side of the modified form become 1-(1-p)^M instead of M*p.

Wouldn't that be a still correct but tighter bound?

Thanks!
Reply With Quote
  #2  
Old 07-12-2012, 10:17 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Hoeffding's inequality with M

Quote:
Originally Posted by ESRogs View Post
Hi,

I'm wondering why in Lecture 2 -- when we're trying to modify the right hand side of Hoeffding's inequality to take into account drawing from multiple bins (see page 16 of the slides) -- we sum all the probabilities of samples being non-representative, rather than taking 1 - the product of all the probabilities of each sample being representative.

In other words, (calling the right hand side of the original inequality p for simplicity) why doesn't the right hand side of the modified form become 1-(1-p)^M instead of M*p.

Wouldn't that be a still correct but tighter bound?
Nice thought. Multiplying would necessitate an independence assumption. While such independence can be easily satisfied in a bin experiment by making sure that different samples are picked independently of each other, the learning counterpart will not follow suit because the different samples are really the same sample (training set) evaluated with different hypotheses.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #3  
Old 07-12-2012, 11:35 PM
ESRogs ESRogs is offline
Member
 
Join Date: Jul 2012
Posts: 12
Default Re: Hoeffding's inequality with M

Oh, I see, that makes sense. The union bound just feels very loose to me, so I was hoping for a tighter one. But it sounds like maybe that's the best we can do without any additional assumptions about the hypothesis set -- is that correct?

Also looking forward to finding out how infinite hypothesis sets are dealt with in future lectures...
Reply With Quote
  #4  
Old 07-12-2012, 11:55 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Hoeffding's inequality with M

Quote:
Originally Posted by ESRogs View Post
The union bound just feels very loose to me, so I was hoping for a tighter one. But it sounds like maybe that's the best we can do without any additional assumptions about the hypothesis set -- is that correct?
Correct. The theory will use a simple property of the hypthesis set to get a much better bound. Stay tuned!
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:32 PM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.