rhm, on 2011-January-07, 10:03, said:
AWM claimed to have done a simulation. Different to me he did not specify his assumptions, he did not say how many deals he generated and looked at. This is very dubious to start with.
I agree with Rainer that providing the simulation results alone is of doubtful value. If you don't provide the details of both the method used and of any subjective decisions made, it makes it hard for people to reproduce your results, or to judge how reliable they are.
If your simulation involves no manual intervention, it's sufficient to provide the constraints (though I usually include the actual code), because that's enough to make it reproducible and criticisable. If, however, you make a subjective decision about each hand, you should provide actual the hands, so that other people can decide whether they agree with your decision.
Look what happened here: for both simulations we were given details of the constraints, and very properly both sets of constraints were discussed and criticised.
Rainer used double-dummy analysis, and some people questioned the reliability of double-dummy analysis for this purpose, so he directed them to an analysis of the consequences of using double-dummy analysis. We can now factor that into our evaluation of Rainer's simulation.
Adam used his judgement, but didn't show us the hands, didn't tell us how many hands he looked at, and didn't tell us what proportion of the hands favoured each action. Hence we have no idea whether there was any inadvertent bias in his analysis, or how accurate his analysis was, or whether he analysed enough hands to make his results reliable.
Quote
Since he claimed he looked at them by hand, hopefully carefully, it can not have been many for the reasons above. This method is far more subjective and biased than large sample double dummy analysis by software.
Even if his analysis by hand is correct, which is difficult to accomplish, and the few deals have been generated randomly according to the specifications, the margin of error will still be high.
It may or may not be more reliable than your double-dummy analysis - it depends upon how many hands he analysed, how good his analysis is, and how strong the effects he noticed were.