Quote:
Originally Posted by Greg Robel
Would someone please clarify for me the first guideline?
"Repeat the experiment for as many runs as needed to get the answer to be stable (statistically away from flipping to the nearest competing answer)."
Thanks!

Hi,
Due to the builtin randomness, you will get different values in different runs. If we express the value in each run as a mean plus noise, where the noise fluctuates around zero with
variance, we need to estimate the mean reliably since this is the value we are after. We can do that by repeating the experiment for many (independent) runs and averaging. Now, because the question is multiplechoice, we only need to make sure that our estimate reliably tells us which of the given options is correct. The quoted guideline is meant to achieve that. If you do say 10,000 runs, you can estimate
per run, conclude that the standard deviation of the average is
and ensure that your answer is correct by checking that you are several standard deviations away from flipping to another of the multiplechoice options.