View Single Post
Old 07-12-2012, 10:17 PM
yaser's Avatar
yaser yaser is offline
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,478
Default Re: Hoeffding's inequality with M

Originally Posted by ESRogs View Post

I'm wondering why in Lecture 2 -- when we're trying to modify the right hand side of Hoeffding's inequality to take into account drawing from multiple bins (see page 16 of the slides) -- we sum all the probabilities of samples being non-representative, rather than taking 1 - the product of all the probabilities of each sample being representative.

In other words, (calling the right hand side of the original inequality p for simplicity) why doesn't the right hand side of the modified form become 1-(1-p)^M instead of M*p.

Wouldn't that be a still correct but tighter bound?
Nice thought. Multiplying would necessitate an independence assumption. While such independence can be easily satisfied in a bin experiment by making sure that different samples are picked independently of each other, the learning counterpart will not follow suit because the different samples are really the same sample (training set) evaluated with different hypotheses.
Where everyone thinks alike, no one thinks very much
Reply With Quote