View Single Post
Old 02-17-2013, 04:08 PM
Haowen Haowen is offline
Join Date: Jan 2013
Posts: 24
Default Re: weight decay and data normalization *not a homework question*

Ok, I understand now. Thank you.

Including w_o captures the practitioner's heuristic "guess" that hypothesis that are closer to zero are "simpler". So the check is, very loosely, that the above assumption is true. Making all the features have zero mean is probably sufficient (in many applications) for the assumption to be "reasonable". However, it is not strictly related since the assumption could hold (or not) for other reasons depending on the application.

In either case, performing validation should allow us to narrow down the types of regularizers that make sense for a particular data set and application.
Reply With Quote