LFD Book Forum  

Go Back   LFD Book Forum > Book Feedback - Learning From Data > Appendix and Notation

Reply
 
Thread Tools Display Modes
  #1  
Old 09-07-2012, 12:03 PM
fgpancorbo fgpancorbo is offline
Senior Member
 
Join Date: Jul 2012
Posts: 104
Default Discussion of the VC proof

Quote:
Originally Posted by admin View Post
For the courageous souls who read the VC proof in the Appendix carefully, please use this subforum to post any questions or comments.

In particular, we will appreciate any comments that help us and other instructors decide whether to include the formal proof in our classroom presentations, or settle for the sketch of the proof that is given in Chapter 2.
After I am done with the final, I'll go through the proof in detail and will let you know what I think. As the course comes to an end, I have nothing but gratitude to you, the Caltech staff and Caltech for making this class available to everybody for free. In addition, it's obvious why you got all those awards for excellence in teaching. I wholeheartedly agree with your approach to teaching the class, especially the selection of the content.
Reply With Quote
  #2  
Old 09-11-2012, 09:08 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,474
Default Re: The VC Proof

Quote:
Originally Posted by fgpancorbo View Post
After I am done with the final, I'll go through the proof in detail and will let you know what I think. As the course comes to an end, I have nothing but gratitude to you, the Caltech staff and Caltech for making this class available to everybody for free. In addition, it's obvious why you got all those awards for excellence in teaching. I wholeheartedly agree with your approach to teaching the class, especially the selection of the content.
Thank you kindly.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #3  
Old 09-18-2012, 07:21 PM
gah44 gah44 is offline
Invited Guest
 
Join Date: Jul 2012
Location: Seattle, WA
Posts: 153
Default Re: The VC Proof

I tend to be closer to an "experimental" scientist. Most often, I don't want to read the proof, though it is nice to know that it is there.

The level covered in the course was just about right for me. I believe I followed it well enough to do the problems, though.

Thanks!
Reply With Quote
  #4  
Old 11-18-2012, 04:29 AM
eakarahan eakarahan is offline
Junior Member
 
Join Date: Nov 2012
Posts: 3
Default Re: The VC Proof

Quote:
Originally Posted by admin View Post
For the courageous souls who read the VC proof in the Appendix carefully, please use this subforum to post any questions or comments.

In particular, we will appreciate any comments that help us and other instructors decide whether to include the formal proof in our classroom presentations, or settle for the sketch of the proof that is given in Chapter 2.
First comment about the proof in Appendix: I experienced a notation problem with P(x, y). This has two meanings. Joint probability or probability density. I would prefer another notation for density, since using one notation with two meanings is ambiguous.

For your question I would like to say: Due to the mixed mathematical background of any classroom, include the formal proof in your classroom presentations. Further, I suggest that you extract the proof from the appendix and include it in the text in Chapter 2, since I think it would be more natural that you develop VC-bound within the lecture formally.

I never like God send formulas, I like to understand the natural development of these ideas and learn how mathematicians transform these ideas into the formulas without any break. A deferred proof, break this sequence force us to remember what was referring what.

If you think the opposite, than enhance your Appendix so that the derivations are given step by step with helping explanations beyond the formulas. Current format of the proof requires the student create this sequence by her/himself. I want to know what is the natural historical development sequence of these ideas. Maybe I should refer the VC-book, Statistical Learning Theory. But, I need the help of an instructor like you, since reading a 750 pages book is infeasible; especially if you yearn for writing your first ML applications within the next 3 months.

For human being english is always better then notations, but notations are inevitable to avoid ambiguities; thus a combination of both is the ideal format. I see you try to achieve this ideal. This is good, so.

At the moment that I wrote this message, I've watched the 6. video and read the book until Chp 2, section 2.3 and I read the two pages of the proof in appendix. Until now, two thinks broke me disturbing my mind; how the Hoeffding found his from God send inequality, and how can I understand the long derivations of VC-Bound including the mathematical technicalities you mention in the book and videos. I'm still working on the second one. And try to accept Hoeffding's inequality as given (which disturbs my mind like a bug in my brain and decelerates my learning).

I know you want to show us the forest instead of dealing with the leaves of trees, but you can do this like you have done in the book between th pages 46-49 using "safe skip" blocks.

A last word, this is the first ML course that I could see the whole picture, the forest. As such this is the first successful attempt that I experienced.

Thank you all, for your effort.
Reply With Quote
  #5  
Old 01-01-2013, 10:26 PM
fgpancorbo fgpancorbo is offline
Senior Member
 
Join Date: Jul 2012
Posts: 104
Default Re: The VC Proof

Well, although with a bit of delay , I did go over the proof. Sorry that it took this long but after the class ended over the summer, I became increasingly busy and I seemed to never find the time to read the proof.

I didn't understand each and every detail but I did understand the overall proof. I think that the level of rigor is about right, given that it's an optional section destined to those who are more mathematically inclined, so I wouldn't change that. Since the most ingenious trick is the idea of approximating E_{out}(h) with E_{in}^{'}(h) for a second data set, my only suggestion is to make the connection that that's exactly what we do in the homeworks, where we put that intuition to work. We estimated E_{out}(h) out of a second set of randomly generated samples, which, if I understood well, is exactly what the trick is (then we averaged out over several runs which is also something that the proof does). The rest of the proof involves the introduction of several technical steps to reach to the final result, but I think that a correct understanding of Lemma A.2 is the key to understanding the overall proof. Since by the end of the course, the student has put the trick to work many times, it relates beautifully a theoretical derivation with the practical work done in the homeworks.

PS1: Of course, I wish a Happy and Prosperous New Year to everyone!
PS2: I was in LA during Christmas and I paid a visit to the Caltech campus. I would have loved to say hello, but since the campus was pretty much empty I assumed that there would be nobody to say hello to .
Reply With Quote
  #6  
Old 01-03-2013, 10:09 AM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,474
Default Re: The VC Proof

Quote:
Originally Posted by fgpancorbo View Post
I was in LA during Christmas and I paid a visit to the Caltech campus. I would have loved to say hello, but since the campus was pretty much empty I assumed that there would be nobody to say hello to .
Thank you for visiting. Indeed, the campus was closed between Christmas and New Year's, and I was traveling at the time.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #7  
Old 01-03-2013, 09:44 PM
fgpancorbo fgpancorbo is offline
Senior Member
 
Join Date: Jul 2012
Posts: 104
Default Re: The VC Proof

Quote:
Originally Posted by yaser View Post
Thank you for visiting. Indeed, the campus was closed between Christmas and New Year's, and I was traveling at the time.
Since I couldn't visit JPL this time, I will probably come back to Pasadena at some point, so maybe next time . Anyhow, it was great to put my feet on campus since studying at Caltech (instead of Stanford) is what I had dreamed of one day doing when I watched Carl Sagan's documentaries as a teen. I was able to take a very good Caltech class virtually, thanks to you. And you are such a terrific teacher that this was a great experience.
Reply With Quote
  #8  
Old 08-09-2013, 01:27 PM
Averroes Averroes is offline
Junior Member
 
Join Date: Aug 2013
Posts: 1
Default Re: The VC Proof

I am a machine learning practitioner currently applying machine learning to algorithmic trading, yet highly interested in the theoretical grounds of the field.

I have read your book "Learning from data" from cover to cover. I haven't solved the problems though. I however did go through the proof of the VC bound in the appendix. I succeeded to understand most of it except (A.4) in the bottom of page 189. I can understand that you have applied Hoeffding Inequality to h*, but your explanation on how this applies to h* conditioned to the sup_H event, is hard to grasp for me.

Can you please give more explanation on how using Hoeffding (A.4) holds ? Or give a reference that helps clarifying this result?

Thanks.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 08:58 PM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.