Question 12

[Anton Khorev]

Looks like there's two answers for Q13. It's possible to get different number of support vectors with octave qp and libsvm.

[yaser]

Originally Posted by Anton Khorev

Looks like there's two answers for Q13. It's possible to get different number of support vectors with octave qp and libsvm.

Interesting. Is the hypothesis identical?

[MLearning]

I think it has to do with the fact that qp ( and quadprog in MATLAB) provide alpha values that are negligbly small. By setting an appropriate threshold, it is possible to filter out these very small values.

In Homework 7, one of the students introduced a trick as means to go around the initialization problem in qp (or quadprod). When I applied this trick, qp and libsvm provide different number of SVs. However, when I initialize all alphas to a vector of zeros, libsvm and Octave's qp yield the same number of SVs.

[Anton Khorev]

In this problem vectors are placed symmetrically. In qp solution one of them touches the margin with alpha==0.

[MLearning]

Originally Posted by Anton Khorev

In this problem vectors are placed symmetrically. In qp solution one of them touches the margin with alpha==0.

Symmetric in X space, yes. How about in Z space, are they symmetric?

[patrickjtierney]

This is the only question I got wrong on the final, and I would have got it right if I used my libsvm version of the answer rather than my hand-built version with qp (all in Octave). My qp (wrong!) answer was one less support vector than I got with libsvm and that might only be because I used 10e-012 as a threshhold. (If I had omitted the threshhold I would have gotten the same number of sv's as in libsvm ).

I got w = [-0.88889, 5.0e-016] and b = -1.6667 using qp, but strangely I get
w = [0.88869, 0] and b = 1.6663 using libsvm. They both have Ein=0 and on a thousand test runs of a million random points in [-3,3]^2 they agree on labels on average 99.999% of the cases. (For libsvm, I use svmpredict with all labels = +1 which is ~71% accurate to get the actual prediction labels.)

The difference in sign may not be significant. I got w and b for qp directly by following the class slides, but I got w = model.SVs'*model.sv_coef and b = - model.rho in the libsvm case (which may not be exactly correct).

The values of alpha (for qp) are different from model.sv_coef, and the qp version uses all but the last of the libsvm support vectors.

So I do agree that there may be 2 correct answers for this question, based on numerical issues and different ways qp and libsvm handle the calculations, but beyond the control of the student.

If required I can PM the alphas and the code I used to support the claim, or wait and post an **answer** after the deadline.

[Anton Khorev]

Originally Posted by yaser

Interesting. Is the hypothesis identical?

Predictions of qp and libsvm are identical (for 10000 uniformly distributed samples on x1,x2 = [-5,5]).