Why your backtest lied to you.
A strategy that looks flawless on historical data and falls apart the moment it's live isn't unlucky. It's one of four mechanisms — and you can test for every one of them before you risk a cent. Here's each, in plain English, with the proof.
You only saw the winners.
The backtest quietly drops everything that died: delisted coins, blown-up accounts, callers who went quiet. What's left looks like skill. Survivorship bias alone inflates apparent returns by 1–4% a year, compounding.
- The check
- Include the dead. Test on the full universe as it existed then, not the survivors that exist now.
Source: Survivorship bias in backtesting (LuxAlgo).
It secretly used the future.
Tune enough knobs and any curve fits the past perfectly. Worse, a single accidental peek at future data turns noise into a flawless equity line. Under realistic conditions, over-tuned strategies have negative expected returns out-of-sample, not zero. The bar for a believable signal is a t-stat above ~3.0, not the textbook 2.0; and even genuine anomalies earn 26% less out-of-sample, 58% less once published.
- The check
- Walk-forward, or true out-of-sample. Decide on data up to now, apply to the next slice you never touched. If the edge evaporates, it was a fit, not a find.
Source: Bailey/Borwein/López de Prado/Zhu (AMS); Harvey/Liu/Zhu (NBER); McLean & Pontiff (J. Finance).
Fees ate the edge before you did.
Backtests show gross returns. Live, every round trip pays the spread, fees, and slippage. Retail round-trip costs run about 7–13 bps; most “edges” sit underneath that. In one study of 4,879 “risk-free” triangular-arbitrage opportunities on Binance, none survived fees.
- The check
- Multiply your spread-plus-fees by your trade count before you celebrate the win rate. If the edge per trade is smaller than the cost per trade, you're paying to lose.
Source: Triangular arbitrage exploitability (ScienceDirect).
One bad day owns the whole book.
A 90% win rate feels safe and hides a fat-tailed loss that eventually arrives. Win rate isn't profit, expectancy is. Grid and martingale “set-and-forget” bots are the purest version: they win small for months, then a single fat-tailed move detonates the whole ladder. Crypto's tails make that move a when, not an if.
- The check
- Expectancy = (win-rate × avg-win) − (loss-rate × avg-loss). Size for the worst day in the data, not the average one.
Source: Gambler's ruin; BTC/ETH tail behaviour (arXiv).
We didn't read this in a book.
Our own HYPE-long bot won most of its trades and still lost money: −31.9% versus −11.5% from simply holding. Win rate is vanity. We ran into all four mechanisms — 27 times — and every one is in the graveyard, dated.