blog
the overfitting verdict belongs at the top of the backtest, not the bottom
There is a specific moment, watching a backtest complete, when a trader can be pulled two ways. On one side is the equity curve — the soaring line that says the last five years would have made you rich. On the other is a stats table with numbers a serious trader knows to interrogate: sample size, out-of-sample separation, drawdown clusters, concentration.
Most backtesting products put the equity curve first and the stats table second. That's the wrong order. It's the order that sells subscriptions, not the order that keeps traders solvent. It is the order that produced most of the retail F&O blow-ups of the last three years.
four tests, before the curve
These are the tests we run on every backtest, and whose result renders as a plain-language verdict at the top of the result page — not buried in a stats table.
- In-sample vs out-of-sample. We split the data by default: the strategy sees the first 70% of history to tune against, and the last 30% only to be evaluated on. If out-of-sample Sharpe is less than half of in-sample, we say so, loudly.
- Trade-count sufficiency. A strategy with 12 trades has metrics that are statistically meaningless. We compute a confidence interval on the win rate given N trades, and if the CI spans losing money, we flag the whole result as "insufficient evidence."
- Concentration. If more than 40% of the returns came from a single week, or more than 60% from a single quarter, we tell you. Most "excellent" backtests are one lucky March away from being unremarkable.
- Instrument fragility. If the strategy backtest is run on a universe but only two instruments produced its edge, we split the equity curve by instrument and show you the divergence. The strategy is fragile in a way the aggregate hid.
the verdict is a sentence
A verdict is not a colour-coded meter. It is one sentence that a trader can read and decide from. "your CAGR looks strong but 62% of it came from a single week in March 2022 — this strategy is fragile to that pattern of vol." "your out-of-sample Sharpe is 30% of your in-sample; treat the backtest as tuned to noise, not signal." That sentence is at the top of the result, above the curve, above every number that flatters.
why every retail tool avoids this
Because it hurts the funnel. A backtest that says "insufficient evidence" is a moment where the user closes the tab. A pristine equity curve is a moment where the user reaches for a broker credential.
We think the tab-close is a feature. A trader who reads honest overfitting warnings and improves their strategy is a trader we keep for years. A trader who deploys a fragile strategy on our platform and blows up in month two is a churn we do not survive.
The Chanel rule works in reverse for backtests: before you show the result, remove one flattering number. If the result stops looking impressive when you do, that is the truth of the result.