 LFD Book Forum Paradox in VC dimension

#1
 PCdimension Junior Member Join Date: Dec 2017 Posts: 1 Paradox in VC dimension

Lets say two reserachers are locked up in 2 separate rooms provided with the same training set. One smart researcher (A) learnt about neural network and SVM, the other B only know about neural network. Lets say the ultimate truth is, neural network is the best model for this particular learning problem and both research A and B submitted the same neural network model.

B happen to have a smaller VC dimension than A as B has a smaller hypothesis test, but both end up choosing the same neural network model as a result.

It looks paradoxical that the less educated researcher B submitted a better model (less VC dimension and requires less number of sample).
______________
Another scenario is that a researcher C had developed a great algorithm to solve a particular learning problem. Later years, countless number of researchers had tried different models but all failed to improve the learning performance. Now the learning problem has grown its VC dimension over time as the total hypothesis space increase. Practically as time pass, the hypothesis will grow to infinity. These all sound paradoxical.

How can we charge for the VC dimension accordingly?
#2
 Burine Junior Member Join Date: Dec 2017 Posts: 1 Re: Paradox in VC dimension

Quote:
 Originally Posted by PCdimension ...both research A and B submitted the same neural network model. It looks paradoxical that the less educated researcher B submitted a better model (less VC dimension and requires less number of sample).
B's model cannot be better, because A and B used the same model.

If you mean B happened to choose a simpler network than A did (e.g. less layers), then since A is more educated clearly he would know how to put weights regularization, dropout,... to avoid overfitting.
______________
Quote:
 Originally Posted by PCdimension Later years, countless number of researchers had tried different models but all failed to improve the learning performance.
On which ground could we assume that countless other models failed while C's model is sub-optimal?
#3
 Yew Lee Junior Member Join Date: Jan 2019 Posts: 3 Re: Paradox in VC dimension

Quote:
 B's model cannot be better, because A and B used the same model. If you mean B happened to choose a simpler network than A did (e.g. less layers), then since A is more educated clearly he would know how to put weights regularization, dropout,... to avoid overfitting.
I think what PCdimension meant is that since researcher (A)'s hypothesis set contains 2 different models, the VC dimension will be larger. Therefore although both researchers submitted the same model, the out of sample error bound for researcher (B) is smaller than that of researcher (A).

 Tags paradox, vc dimension Thread Tools Show Printable Version Email this Page Display Modes Linear Mode Switch to Hybrid Mode Switch to Threaded Mode Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 02:18 AM.