LFD Book Forum  

Go Back   LFD Book Forum > Book Feedback - Learning From Data > Chapter 4 - Overfitting

Reply
 
Thread Tools Display Modes
  #1  
Old 10-23-2013, 12:15 PM
Sweater Monkey Sweater Monkey is offline
Junior Member
 
Join Date: Sep 2013
Posts: 6
Default Exercise 4.7

I feel like I'm overthinking Exercise 4.7 (b) and I am hoping for a little bit of insight.

My gut instinct says that Var[E_{\text{val}}(g^-)] = \frac{1}{K} (P[g^-(x),y])^2

I arrived at this idea by considering that the probability is similar to the standard deviation which is the square root of the variance so since:
Var[E_{\text{val}}(g^-)] = \frac{1}{K}Var_{x}[e(g^-(x),y)] and P[{g^-(x)}\neq{y}] = P[e(g^-(x),y)] does Var_{x}[e(g^-(x),y)] = P[{g^-(x)}\neq{y}]^2 ???

Then for part (c) on the exercise, assuming that the above is true, I used the notion that P[{g^-(x)}\neq{y}] \le 0.5 because if the probability of error were greater than 0.5 then the learned g would just flip its classification. Therefore this shows that for any g^- in a classification problem,
Var[E_{\text{val}}(g^-)] \le \frac{1}{K}0.5^2 and therefore:
Var[E_{\text{val}}(g^-)] \le \frac{1}{4K}

Any indication as to whether I'm working along the correct lines would be appreciated!
Reply With Quote
  #2  
Old 10-25-2013, 08:16 AM
magdon's Avatar
magdon magdon is offline
RPI
 
Join Date: Aug 2009
Location: Troy, NY, USA.
Posts: 592
Default Re: Exercise 4.7

Quote:
Originally Posted by Sweater Monkey View Post
I feel like I'm overthinking Exercise 4.7 (b) and I am hoping for a little bit of insight.

My gut instinct says that Var[E_{\text{val}}(g^-)] = \frac{1}{K} (P[g^-(x),y])^2
Var[E_{\text{val}}(g^-)] can be obtained from part (a) by computing \sigma^2(g^-), which is the variance (over x) of the error that the hypothesis makes. For the specific error measure, the error is bounded between [0,1], so you can bound this variance.
__________________
Have faith in probability
Reply With Quote
  #3  
Old 04-06-2016, 03:51 AM
ntvy95 ntvy95 is offline
Member
 
Join Date: Jan 2016
Posts: 37
Default Re: Exercise 4.7

Hello, I'm currently stuck at (d). I have derived to this point:

Var_{x}[(g^{-}(x) - y)^{2}] = E[((g^{-}(x) - y)^{2} - E[(g^{-}(x) - y)^{2}])^{2}]
E[((g^{-}(x) - y)^{2} - E[(g^{-}(x) - y)^{2}])^{2}] = E[(g^{-}(x) - y)^{4}] - (E[(g^{-}(x) - y)^{2}])^{2}

and I'm currently stuck. The hint says that the squared error is unbounded hence I guess that there should be no bound for expected value of squared error? (I'm not good at math though...)
Reply With Quote
  #4  
Old 07-19-2016, 09:58 AM
ntvy95 ntvy95 is offline
Member
 
Join Date: Jan 2016
Posts: 37
Default Re: Exercise 4.7

I'm not sure if I can re-interpret the Figure 4.8 like this: If you train your data with one horrible hypothesis you will get a very bad generalization bound despite the number of data points is large?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 03:33 AM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.