*ANSWER* questions w linear regression & weight decay

[Michael Reach]

I have been running the weight decay examples Q2-6, but haven't seen any real improvement in the out-of-sample error compared to no regularization at all. Is that just a feature of this particular problem, or should I recheck my calculations?

Unfortunately (or not), the answers I've been getting do appear as options on the multiple choices.

[Ziad Hatahet]

You should be seeing a change in the out-of-sample error when you vary (for certain values of at least). Are you using classification error as your error measure?

[jlaurentum]

Are you using the correct formula for the one step solution?

I was using instead of and so regularization didnt make sense at all. I caught the error because I saw in another post that Professor Yaser corrected a student on the plus sign.

[Michael Reach]

Originally Posted by Ziad Hatahet

You should be seeing a change in the out-of-sample error when you vary (for certain values of at least). Are you using classification error as your error measure?

Well, I am seeing a change, just not really a reduction. Some are the same, some are bigger. I was expecting a dramatic drop in the out-of-sample error.

And yes, I have been using classification error, but that is a good point - I started using the regression residuals and such, but that mistake at least I caught.

[Michael Reach]

As I suspected, all my answers on these were wrong. Does anyone have code (R if possible) to show, that I could use for comparison? I'm suspecting my problem was something dumb; even the original linear regression was wrong, and I compared that one with the same answer from the R lm() function.

I'm especially concerned since HW 7 uses all the same data again - so I really need to track this down.

[Elroch]

Originally Posted by Michael Reach

As I suspected, all my answers on these were wrong. Does anyone have code (R if possible) to show, that I could use for comparison? I'm suspecting my problem was something dumb; even the original linear regression was wrong, and I compared that one with the same answer from the R lm() function.

I'm especially concerned since HW 7 uses all the same data again - so I really need to track this down.

Are you using lambda from 0.001 to 1000? I suppose it might be possible to forget to calculate the power. If you do this, the added term in the matrix described in this thread can hardly fail to have a significant effect.

[jlaurentum]

Michael:

Here you go:

#READ IN THE FILES.
datos1 <- read.table("in.dta")
names(datos1) <- c("X1","X2","Y")
datos2 <- read.table("out.dta")
names(datos2) <- c("X1","X2","Y")
#FOR THE FOLLOWING QUESTIONS, SET UP THE MATRIXES
Z <- with(datos1,
			cbind(rep(1,nrow(datos1)),X1,X2,
						X1^2,X2^2,X1*X2,abs(X1-X2),abs(X1+X2)) )
Z <- as.matrix(Z)
Zout <- with(datos2,
					cbind(rep(1,nrow(datos2)),X1,X2,
						X1^2,X2^2,X1*X2,abs(X1-X2),abs(X1+X2)) )
Zout <- as.matrix(Zout)
#NOW FIT WITH WEIGHT DECAY USING LAMBDA=10^-3
lambda <- 10^(-3)

M <- t(Z)%*%Z + diag(rep(8,1))*lambda
w <- solve(M)%*%t(Z)%*%datos1$Y
Ym <- as.numeric(sign(Z%*%w))
Ein <- mean(datos1$Y!=Ym)
Ym <- as.numeric(sign(Zout%*%w))
Eout <- mean(datos2$Y!=Ym)