LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Homework 6 (http://book.caltech.edu/bookforum/forumdisplay.php?f=135)
-   -   Subset & deterministic noise (http://book.caltech.edu/bookforum/showthread.php?t=1075)

apinde 08-21-2012 09:06 AM

Subset & deterministic noise
 
Related to Q1 in HW6 I used the following reasoning to say that there is no trend. Whats the falacy?

Construct the hypothesis set H as the union of two hypothesis sets H1 and H2
Let H1 have higher deterministic noise and H2 have lower
H will have deterministic noise between H1 and H2
Now let H' be the set H1. Clearly H' is a subset of H ans H' has higher deterministic noise than H
Similarly H' can be H2, and then H' has lower deterministic noise than H

Therefore there is no trend

Andrs 08-21-2012 09:44 AM

Re: Subset & deterministic noise
 
If H is the union of H1 and H2 and assuming that H1 differs from H2, the hypothesis set H should be at least as complex as H2 therfore it should have a lower deterministic noise.
Related to the question:
The basic charachteristic is that H' is a subset of H therefore H' contains fewer alternatives compared to H (less options less flexibility). That is, H' will not be able to fit into the target function f as well as H would have. That is, H' implies a higher "deterministic noise".

apinde 08-21-2012 10:42 AM

Re: Subset & deterministic noise
 
Thanks for your response, Andrs.

Another expression of deterministic noise is the expected value of the squared error between the mean hypothesis and f(x). The mean hypothesis when one has disjoint H1 and H2 should fall somewhere between the more complex and the more simple sets and the deterministic error should be somewhere between the deterministic errors of H1 and H2. It other words, the deterministic error of H is not just the lower of the two.

MLearning 08-21-2012 11:21 AM

Re: Subset & deterministic noise
 
The argument that there is no trend seems to make sense to me. We are not told about the complexity of the hypothesis, what we are told is both H' and H contain a set of hypothesis functions which can be simple or complex.

I disagree with the assertion that fewer alternatives means less options and less complexity. A hypothesis set may have a few number of possible hypothesis functions but one of those functions can have the complexity to provide a good fit. Hence, less deterministic noise.

MLearning 08-21-2012 11:40 AM

Re: Subset & deterministic noise
 
The argument that there is no trend seems to make sense to me. We are not told about the complexity of the hypothesis, what we are told is both H' and H contain a set of hypothesis functions which can be simple or complex.

I disagree with the assertion that fewer alternatives means less options and less complexity. A hypothesis set may have a few number of possible hypothesis functions but one of those functions can have the complexity to provide a good fit. Hence, less deterministic noise.

JohnH 08-21-2012 12:30 PM

Re: Subset & deterministic noise
 
The phrase "deterministic noise" refers to complexity in the target function that cannot be described by the hypothesis set. \mathcal{H} is analogous to a vocabulary with each h being a word within that vocabulary. Removing words from the vocabulary, \mathcal{H}'\subset\mathcal{H}, diminishes expressive capability; i.e., there are fewer things that can be described.

MLearning 08-21-2012 12:39 PM

Re: Subset & deterministic noise
 
@JohnH,

I see your point and your argument makes sense. However, what if those remaining volcabularies in H' are complex enough to completely characterize the target function. Remember the target function is fixed in this case.

JohnH 08-21-2012 03:13 PM

Re: Subset & deterministic noise
 
The question was about trending, not a specific case. The question could be posed in terms of expected outcome. Is \mathcal{H}' more or less likely than \mathcal{H} to be sufficient to describe the target function?

ilya239 02-13-2013 07:00 PM

Re: Subset & deterministic noise
 
Quote:

Originally Posted by MLearning (Post 4227)
@JohnH,

I see your point and your argument makes sense. However, what if those remaining volcabularies in H' are complex enough to completely characterize the target function. Remember the target function is fixed in this case.

Deterministic noise will either be reduced or stay the same. So the change is zero with some probability and negative with some probability, so the expected change is negative. (What a hand-waving argument :) )


All times are GMT -7. The time now is 11:46 AM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.