Backtesting does not prove that future returns will be anything like past returns. However it is the best tool we have and is significantly better than just arbitrarily picking some parameter settings and going with them.
"with back testing you know what the stock is going to do and you can fiddle with the knobs to optimise for the dips and crests."
What you've described is known as overfitting. If you backtest improperly then your tests are quite useless because anyone can come up with parameter settings that maximize returns for a given set of data. However these parameters probably won't do well with the majority of new data. And that's the key to backtesting.
You MUST divide your data into two parts: "in sample" data and "out of sample" data. You use in sample data to optimize your parameters (in AI 2.0 the process of optimizing parameters is called "Model Building") and then test your results on the out of sample data. If the results are similar to what you saw with your in sample data, then you should feel quite good about your model (i.e. your model has been confirmed).
However if your out of sample test is significantly different from your in sample test, then you would have a low level of confidence about your model. The other important fact is that the more data you use to build your models and the more market types they include (i.e. Bear, Bull, sideways), the better your model will be.
Finally you should calculate the statistical error in your results.
To sum up, while backtesting does not guarantee great results, it certainly increases the probability of seeing great results over simply setting parameters based on plucking numbers out of the air or going with what somebody told you.
News 
Market Data 
Discover 
AI
