Thread: Hw 6 q1
View Single Post
Old 05-14-2013, 04:33 PM
Elroch Elroch is offline
Invited Guest
Join Date: Mar 2013
Posts: 143
Default Re: Hw 6 q1

Originally Posted by yaser View Post
I understand how the cases where f was excplicitly given can cause confusion, as this seems to go against the main premise of unknown target function. The best way to resolve this confusion is to assume that someone else knows what the target is, and they will use that information to evaluate different aspects of our learning process, but we ourselves don't not know what f is as we try to learn it from the data.

Having said that, the notion of 'fixed' is different. Q1 describes two learning processes (with two different hypothesis sets) and asserts that both processes are trying to learn the same target. That target can be unknown to both of them, but it is the same target and that is what makes it fixed. The point of having f fixed here is that deterministic noise depends on more than one component in a learning situation, and by fixing the target function we take out one of these dependencies.
It has indeed been a source of some discomfort that the phenomenon being studied depends on something that is fixed but unknown! As far as I can see, it is possible to be given the same data and use the same method, and to be overfitting with one target function, but underfitting with another.

This is what made me think when I first saw this issue that it was necessary to have some knowledge about the distribution of the possible functions in order to allow the possibility of assessing the quality of a particular machine learning algorithm for function approximation in a real application. However, I now believe that using the technique of cross-validation gives an objective way of studying out of sample performance for function approximation that should allow probabilistic conclusions roughly analogous to Hoeffding. [I am familiar with this technique from the optimization of hyperparameters when using SVMs]

One of the great things about doing this course is to get to grips with issues like this. In fact I was using the C hyperparameter without really knowing what it was before we got to regularization in the lectures! I hope I've got the right end of the stick now.
Reply With Quote