Subset & deterministic noise

[apinde]

Related to Q1 in HW6 I used the following reasoning to say that there is no trend. Whats the falacy?

Construct the hypothesis set H as the union of two hypothesis sets H1 and H2
Let H1 have higher deterministic noise and H2 have lower
H will have deterministic noise between H1 and H2
Now let H' be the set H1. Clearly H' is a subset of H ans H' has higher deterministic noise than H
Similarly H' can be H2, and then H' has lower deterministic noise than H

Therefore there is no trend

[Andrs]

If H is the union of H1 and H2 and assuming that H1 differs from H2, the hypothesis set H should be at least as complex as H2 therfore it should have a lower deterministic noise.
Related to the question:
The basic charachteristic is that H' is a subset of H therefore H' contains fewer alternatives compared to H (less options less flexibility). That is, H' will not be able to fit into the target function f as well as H would have. That is, H' implies a higher "deterministic noise".

[apinde]

Thanks for your response, Andrs.

Another expression of deterministic noise is the expected value of the squared error between the mean hypothesis and f(x). The mean hypothesis when one has disjoint H1 and H2 should fall somewhere between the more complex and the more simple sets and the deterministic error should be somewhere between the deterministic errors of H1 and H2. It other words, the deterministic error of H is not just the lower of the two.

[MLearning]

The argument that there is no trend seems to make sense to me. We are not told about the complexity of the hypothesis, what we are told is both H' and H contain a set of hypothesis functions which can be simple or complex.

I disagree with the assertion that fewer alternatives means less options and less complexity. A hypothesis set may have a few number of possible hypothesis functions but one of those functions can have the complexity to provide a good fit. Hence, less deterministic noise.

[MLearning]

The argument that there is no trend seems to make sense to me. We are not told about the complexity of the hypothesis, what we are told is both H' and H contain a set of hypothesis functions which can be simple or complex.

I disagree with the assertion that fewer alternatives means less options and less complexity. A hypothesis set may have a few number of possible hypothesis functions but one of those functions can have the complexity to provide a good fit. Hence, less deterministic noise.

[JohnH]

The phrase "deterministic noise" refers to complexity in the target function that cannot be described by the hypothesis set. is analogous to a vocabulary with each being a word within that vocabulary. Removing words from the vocabulary, , diminishes expressive capability; i.e., there are fewer things that can be described.

[MLearning]

@JohnH,

I see your point and your argument makes sense. However, what if those remaining volcabularies in H' are complex enough to completely characterize the target function. Remember the target function is fixed in this case.

[JohnH]

The question was about trending, not a specific case. The question could be posed in terms of expected outcome. Is more or less likely than to be sufficient to describe the target function?

[ilya239]

Originally Posted by MLearning

@JohnH,

I see your point and your argument makes sense. However, what if those remaining volcabularies in H' are complex enough to completely characterize the target function. Remember the target function is fixed in this case.

Deterministic noise will either be reduced or stay the same. So the change is zero with some probability and negative with some probability, so the expected change is negative. (What a hand-waving argument )