Why your backtest lied to you.
A strategy that looks flawless on historical data and falls apart the moment it's live isn't unlucky. It's one of four mechanisms, and you can test for every one of them before you risk a cent. Here's each, in plain English, with the proof.
You only saw the winners.
The backtest quietly drops everything that died: delisted coins, blown-up accounts, callers who went quiet. What's left looks like skill. Survivorship bias alone inflates apparent returns by 1–4% a year, compounding.
- The check
- Include the dead. Test on the full universe as it existed then, not the survivors that exist now.
Source: Survivorship bias in backtesting (LuxAlgo).
It secretly used the future.
Tune enough knobs and any curve fits the past perfectly. Worse, a single accidental peek at future data turns noise into a flawless equity line. Under realistic conditions, over-tuned strategies have negative expected returns out-of-sample, not zero. The bar for a believable signal is a t-stat above ~3.0, not the textbook 2.0; and even genuine anomalies earn 26% less out-of-sample, 58% less once published.
- The check
- Walk-forward, or true out-of-sample. Decide on data up to now, apply to the next slice you never touched. If the edge evaporates, it was a fit, not a find.
Source: Bailey/Borwein/López de Prado/Zhu (AMS); Harvey/Liu/Zhu (NBER); McLean & Pontiff (J. Finance).
Fees ate the edge before you did.
Backtests show gross returns. Live, every round trip pays the spread, fees, and slippage. Retail round-trip costs run about 7–13 bps; most “edges” sit underneath that. In one study of 4,879 “risk-free” triangular-arbitrage opportunities on Binance, none survived fees.
- The check
- Multiply your spread-plus-fees by your trade count before you celebrate the win rate. If the edge per trade is smaller than the cost per trade, you're paying to lose.
Source: Triangular arbitrage exploitability (ScienceDirect).
One bad day owns the whole book.
A 90% win rate feels safe and hides a fat-tailed loss that eventually arrives. Win rate isn't profit, expectancy is. Grid and martingale “set-and-forget” bots are the purest version: they win small for months, then a single fat-tailed move detonates the whole ladder. Crypto's tails make that move a when, not an if.
- The check
- Expectancy = (win-rate × avg-win) − (loss-rate × avg-loss). Size for the worst day in the data, not the average one.
Source: Gambler's ruin; BTC/ETH tail behaviour (arXiv).
We didn't read this in a book.
Our own HYPE-long bot won most of its trades and still lost money: −31.9% versus −11.5% from simply holding. Win rate is vanity. We ran into all four mechanisms, 33 times, and every one is in the graveyard, dated.