[Elroch]

Originally Posted by Michael Reach

Thanks!

Yes, Elroch, I used the full range of lambda. I think my mistake is elsewhere.

If you're like me, you've probably made a silly error which has nothing to do with understanding the method.

ok, I'm going to expose most of my insult to the art of programming for these questions. Don't use it as a style guide (especially that nasty bit of unvectorised code. Also I suspect the as.matrix's may be superfluous.) The data format should be clear, I hope.

WeightDecayLinearRegressionSolver <- function(inputs, outputs, lambda) {  
  # note inputs have bias co-ordinate 
  # inputs is a matrix of 2d points (with a bias)
  # outputs is a vector providing a real valued function of those points
  if (isTRUE(all.equal(var(outputs), 0))) {  
    # This is the completely degenerate case, which occurs when trying to classify data of a single class
    result <- c(outputs[1], 0, 0)
  }
  else {
    result <- PseudoInverse(t(as.matrix(inputs)) %*% as.matrix(inputs) + diag(rep(lambda, length(inputs[1,]))) ) %*% t(as.matrix(inputs)) %*% outputs
  }
  result
}

PseudoInverse <- function(mat) {
  tmat <- t(as.matrix(mat))
  inv(tmat %*% as.matrix(mat)) %*% tmat
}

ClassificationError <- function(actual, predicted) {
  result = 0
  for(i in 1:length(actual)) {
    if(abs(actual[i] - predicted[i]) > 0.5) {
      result <- result + 1
    }
  }
  result/length(actual)
}