Hi Too Fuzzy,
You are addressing a well known issue that has been discussed in general many times. Your argument is quite valid but it does not fully apply to my post you are reacting to.
First let me state that the term "Back Testing" appears to be used in two different ways on the AIM Forum and no doubt I have not always been consistent as I used it too in a way it is probably not meant.
In various posts Back Testing with AIM appears simply to be Running of a set of historical stock prices with a standard parameter set and perhaps a run with another favourite parameter set suggested by others as being good too, and to compare the results with the Buy & Hold Results or to compare the AIM results with the results of another type of programme. . .This was, I believe the type of Back testing that was discussed lately by Ganaraska.
I have exclusively directed my self to the Optimisation of a historical Dataset and to use that Dataset for an actual Investment run. It is clear to all that have participated ion these discussions that these optimised parameters would only be useful if the future data was approximately repetition of the historical Dataset. I have had many discussion with Mark Hing of A.I. on the issue. There is no disagreement on the weak part of Optimisation. Mark and I discussed a method that was potentially useful:
1 Start with optimisation using historical data;
2 Run the real AIM with approximately the optimised data but temper the aggression factors of Vortex somewhat;
3 Run the AIM for say 6 months or so to see how the trend develops, if the Price behaviour is drastically different then the optimised parameters will not be effective;
4 Redo the optimisation after removing a 6 months data at the back-end of the old DataSet and optimise to the current date. This would reset the Parameters with historical data shifted to the latest date. . .Then wait 6 month again and if required repeat the optimisation.
In case the stock behaviour remains “on “ target” then the result will be close to the optimum. . . if not then you keep re-optimising.
In SPY1, using Windows Vortex AIM I had already started on 03-01-2011 using parameters from some manual optimisation of the Historical Dataset Long before that and I AIMed SPY1 for about 8 @ 9 months without changing any parameters. In the mean time we had started developed our Optimisation Program and started trials some times in 2012. Up In 2011 I still used the manual Optimisation method with the Vortex Excel Spread sheet and on that basis I modified the parameters based on a re-optimisation. All the Demo investment were made in Vortex Windows with parameter optimisation done with historical data in Excel. This was exactly the method developed by me in discussion with Mark Hing advice on the generalities that he had already developed himself for AI.
At the end of 2102 and for 2013 I repeated the optimisations about two times but these times with rather short historical periods. This is so because in Vortex I only had downloaded Spy prices since May 2011 and our Optimisation Program can only take prices out of the Vortex Windows Data Bank. So for the Spy1 Run the result are based on optimisations using historical data and the decision that a new optimisation was needed was an Overlay Management Decision. . this is something any Investment manger must do.
IN [b[SPY2 and SPY3 I used a different approach and altered the parameters 2 times. . .beyond what was indicated by the Optimisation Program ( Manager Interference) after the start-up, giving 3 parameter sets for a ~ 2,7 year Run.
This provides some extra information in regards to what I am doing. Most of the content was directed to present our Optimisation program and that I had started using it and that if anyone would be interested they could acquire a copy when they are using Vortex. It is of course clear that the short period I used for the Machine Optimisations that my numerical results may not be demonstrably indicative of a very high yield.
The Vortex SPY ROTAC Yields since January 2011 to now, that lie in the range of 10 to 12 % annually, may well be very similar to the results people on the AIM USER Forum have been getting. It is interesting to know what other AIM Investors have scored with their aim for the same period.
News 
Market Data 
Discover 
