View Single Post
Old 03-04-2013, 08:23 AM
hemphill hemphill is offline
Join Date: Jan 2013
Posts: 18
Default Re: How many support vectors is too many?

Originally Posted by yaser View Post
For an absolute estimate, the theoretical result mentioned at the end of Lecture 14 (last slide) shows the generalization error to be \approx bounded by the ratio between the number of support vectors and the number of examples. For a comparative estimate, the same result implies that the number of support vectors behaves more or less like a VC dimension of other models.

Of course there are situations where neither SVM nor other models will perform to a satisfactory level, as we would expect if the resources of data are not adequate to capture the complexity of the target.

Indeed, the cross validation estimate can overrule (our interpretation of) the theoretical VC bound. The VC is just a bound that applies to all cases, while cross validation provides a concrete estimate for a particular case.
I have noticed that the support vector ratio sometimes provides a very good bound, at least with our toy problems. Suppose we had a situation like:

C=C_1, E_{\rm cv}=0.050, support vectors=20
C=C_2, E_{\rm cv}=0.049, support vectors=60

where the size of the data set is around 400. I would be tempted to choose C_1, regardless of E_{\rm cv}. What are your thoughts?
Reply With Quote