I'll be updating the current research document with Results, Analysis and Conclusions tonight, but here's a preview.
The basic conclusion is going to be that this experimental design was flawed in a way that radically increases the probability of "type-2 errors" or "false negatives".
The problem is in the way I set up the signal group and control group in each set of price bars. The signal group was composed of every bar at which the signal was present, and the control group consisted of every other bar. This is not the right way to proceed because much of the control group is still affected by the signal. Many of the bars in the control group would have occurred right after a signal. Therefore, if the signal really is correlated with a certain price behavior, then the price behavior would be occurring in the vicinity of many of the control bars as well, just because the control bars are near the signal bar.
For example, suppose a positive MA crossover really is correlated with a subsequent uptrend in price. A bar at which a crossover occurred is part of the signal group, and the next few bars would be part of the control group. If the price rises because of the correlation with the signal, that price rise is occurring right after these control bars as well. So when we calculate the mean price rise associated with control bars, the value will be skewed upward. In effect, it would look like price rises are VERY COMMON, and so the price rise associated with the signal bar is no big deal. This isn't true though. If the control bars hadn't been tainted by their proximity to the signal, price rises wouldn't look nearly as common, and the price rise associated with our signal might very well look like a big deal.
I think the solution to this design flaw will lie in combining the ideas of chart "events" and "conditions" in one testing project instead of treating them in two separate projects as is currently shown on the Pending Project List.
As an example, once a moving average crossover EVENT occurs, we have a CONDITION on the chart where price is greater than the MA (or whatever). There is a unique set of bars for which this condition is true, and that set of bars is bounded by two events; the positive crossover at the beginning and a negative crossover at the end. Our signal group and control group should consist of these separate sets of bars.
Once I finish the Results, Analysis and Conclusions for this present project and move it over to the archives, I'll begin work on this modified approach.
Gotta get to work...
