The Polymarket signal we stopped testing, because the answer needs nine years of data
A signal on a prediction-market exchange looked like it might pay, and a month earlier we had said we simply needed more data. So we collected five more weeks, seventeen times as much, and it was still indistinguishable from luck. This time we worked out how much data would actually settle it: about ninety-two thousand trades, arriving at twenty-eight a day, which is about nine years. The trouble is not the amount of data, it is that a few enormous trades swamp everything else. Some questions cannot be answered, and saying so is far cheaper than collecting for a decade to find out.
In July we read this as "underpowered, collect more data". Five weeks and seventeen times the tape later it is still indistinguishable from zero, and this time we did the arithmetic on how much data would actually settle it: about 91,900 trades, arriving at 27.9 a day, so roughly nine years. The obstacle is not the length of the tape, it is the spread of outcomes. A handful of blow-up trades dominates a thousand observations and will dominate ten thousand. "Collect more" was the wrong answer. The honest verdict is that this question cannot be settled inside any horizon that matters, which makes it a closed question rather than an open one, so we stopped paying to keep it open.
Re-read against the full accrued tape: 324 files, 45 distinct days, 2026-06-29 to 2026-08-12, 50,617 markets loaded and 50,613 resolved. The stale-price leg: 1,255 takes, 40.0% win rate, +$0.5012 average net per take, standard deviation $75.97 per take, t=+0.23. Neither pre-committed branch fires, so the sign must not be read as either a kill or a go. The sample required for significance is (2 x 75.97 / 0.5012)², about 91,900 takes, against an observed 27.9 takes a day, which is roughly 3,295 more days of identical collection. Dispersion is the binding constraint, not tape length: these takes carry 7.1 times the per-take variance of the no-skill baseline ($75.97 against $10.68) while being worth +$1.31 a take against it. Size-capping is heavy and disclosed: 493 of the 1,255 takes wanted a clip larger than the entire top of the book. The general rule this produced, now standing: when a gate returns underpowered, compute the required sample and divide it by the observed arrival rate before ever saying "collect more". Five weeks of disk bought a question that could never resolve.
- Kill date
- 2026-08-12
- Sample
- 1,255 takes over 45 days
- Method
- Pre-registered live test
- Verdict
- un-powerable, not unproven
Pre-registered before the data, judged on a criterion locked in advance, and published whatever the result.
See all kills