When a rule stops working
Every rule eventually meets a market it was not built for. How we watch for that moment, and what we do when it comes.
Passing every test does not make a rule permanent. Markets change: participants adapt, volatility shifts, and an effect that was real for years can fade. The work does not end when a rule goes live; it changes from testing to watching.
1. Know what normal looks like
A rule’s test record tells us more than its average result. It also tells us what normal variation looks like: the usual hit rate, the typical size of wins and losses, how long losing streaks tend to last and how deep the worst stretches went. Before a rule trades, we write those ranges down. Live results are then judged against a range, not a single number, because even a healthy rule spends plenty of time below its average.
2. Watch for drift, not noise
The signs that something has changed are usually gradual: a hit rate that sits below its expected band for longer than chance would explain, a drawdown that is deeper or longer than anything in the tested history, trading costs that run consistently above what was modelled, or a rule that suddenly fires at different times or far more or less often than before. Any one of these is a question. Several together are an answer.
3. Small samples lie
The hardest part is patience in both directions. A handful of losing trades says almost nothing about a rule; reacting to them is how good rules get abandoned at the bottom of a normal dip. But waiting forever is its own mistake. The answer is to decide in advance how much evidence is enough, and to respect that threshold even when the most recent trades are loud.
4. Decide before the data arrives
When we test a change, the rule for success is written down before the results come in: how many trades we need, what counts as better, and that it must hold in both halves of the sample. This matters because the most tempting fix, tweaking a rule’s settings until its recent results look good again, is simply overfitting to the newest data. A change has to earn its place through the same tests as the original rule did.
5. Retire without regret
When the evidence says a rule has stopped working, it is reduced or switched off, and its capital goes back to the rules that are still earning their place. A retired rule can come back, but only by passing the full set of tests again. Letting go of a rule that once worked is uncomfortable. Keeping it because it once worked is expensive.
This note describes research methods. It is not investment advice and does not describe the performance of any account or fund.