#1




Paradox in VC dimension
Let’s say two reserachers are locked up in 2 separate rooms provided with the same training set. One smart researcher (A) learnt about neural network and SVM, the other B only know about neural network. Let’s say the ultimate truth is, neural network is the best model for this particular learning problem and both research A and B submitted the same neural network model.
B happen to have a smaller VC dimension than A as B has a smaller hypothesis test, but both end up choosing the same neural network model as a result. It looks paradoxical that the less educated researcher B submitted a better model (less VC dimension and requires less number of sample). ______________ Another scenario is that a researcher C had developed a great algorithm to solve a particular learning problem. Later years, countless number of researchers had tried different models but all failed to improve the learning performance. Now the learning problem has grown its VC dimension over time as the total hypothesis space increase. Practically as time pass, the hypothesis will grow to infinity. These all sound paradoxical. How can we charge for the VC dimension accordingly? 
#2




Re: Paradox in VC dimension
Quote:
If you mean B happened to choose a simpler network than A did (e.g. less layers), then since A is more educated clearly he would know how to put weights regularization, dropout,... to avoid overfitting. ______________ On which ground could we assume that countless other models failed while C's model is suboptimal? 
Tags 
paradox, vc dimension 
Thread Tools  
Display Modes  

