When to stop testing a stock thesis
Stop testing a stock thesis when a predeclared test fails in a way the thesis cannot explain, or when each new run exists only to rescue the last one. Keep testing when a result points to one specific, economically meaningful assumption you can change before looking again. A good next test still asks a real question.
Retiring a thesis is progress. Research should shrink the pile of live rules. A backtest can rule out a story about the market. It can't prove that the surviving story will keep working, and it shouldn't be asked to.
What counts as a failed thesis?
A thesis is a claim with rules attached. "Profitable companies with improving cash conversion outperform" is a claim. The rules might measure profitability with trailing net income, cash conversion with operating cash flow divided by net income, and improvement against the same fiscal quarter a year earlier. The exact proxy matters because the test can only reject the version you wrote down. That's why vague rules make bad tests.
Failure should be declared before the run. Write a sentence such as: I will stop if the rule trails an appropriate benchmark across the development period and the holdout, or if the apparent advantage disappears when I make small, predeclared changes to the thresholds. That is a decision rule. It sets out the evidence that would count against the claim.
A weak result does not automatically mean failure. The available history may be short. Coverage may be thin, or the rule may qualify too few events to say much. Call that inconclusive, preserve the run, and resist turning it into a verdict. A thesis fails when the available evidence contradicts its stated claim. It is inconclusive when the evidence cannot distinguish the claim from noise. Those are different outcomes and deserve different next steps.
Checking data coverage before a backtest comes first for this reason. If missing price history or fundamentals changed the usable universe, a disappointing result might describe the surviving data. Coverage does not excuse a result. It tells you what result you actually have.
Run the test you wrote before changing it
The common escape hatch is familiar. The original rule loses, so you lower a threshold. Then you lengthen the holding period. Then you remove a bad year, narrow the universe, or add a second metric that would have excluded the ugly names. Each edit can sound sensible in isolation. Together they turn the original thesis into a search for a chart you like. You've stopped testing and started shopping.
Do the opposite. Preserve the first version and run the test it earned. Keep the benchmark, rebalance schedule, reporting lag, universe, and holding period written down. A backtest report records rules meeting a historical sample. Change any one of those items and you have a new specification. Name it that in your notes.
A thesis doesn't need to stay frozen forever. The change needs a reason that existed outside the disappointing line chart. Fix a threshold if it was a typo. Replace a field if a filing convention makes it unusable for an industry. If the hypothesis was always about persistent cash generation and the first draft measured one quarter, repair the mismatch. State the reason, change one thing, and label the result as a new thesis version.
The discipline matters most after a failure. A successful result tempts people to celebrate. A bad one tempts them to search. The search is where the false discovery risk lives.
Small changes tell you whether there is anything there
A robust rule survives nearby versions of itself. This is sensitivity testing: moving one reasonable assumption at a time to see whether the conclusion changes. A quality threshold expressed as a round number might deserve a slightly stricter and slightly looser neighbor. A quarterly rebalance might deserve a slower schedule if the economic story says the signal should persist.
You are looking for a pattern that persists across settings you could have defended in advance. Testing whether a backtest result is real gives the same discipline a practical sequence of checks. If only one tiny corner works, that corner is a lead for further skepticism. It isn't a finished rule.
Use sensitivity tests to separate three outcomes:
- The result holds across reasonable choices. Keep researching. The thesis has earned a harder challenge, such as a holdout period or a different market regime.
- The result weakens but keeps its direction. The effect may be small, sensitive to costs, or too weak to justify the complexity. Keep the record, then decide whether the economic story warrants more work.
- The direction flips or the effect vanishes. Retire the version you tested. If there is a new explanation for the instability, write a new hypothesis. Don't pretend the old one survived.
No test can tell you whether a threshold is the one true value. Markets don't have a sign saying that a business becomes attractive at a particular ratio. Sensitivity testing asks a narrower question: did your conclusion depend on knowing that imaginary line with impossible precision?
A holdout result is evidence you cannot reuse
A holdout period is a stretch of history you leave untouched while developing a thesis. You settle the rule using earlier data, freeze it, and then inspect the holdout once. It supplies evidence that didn't choose the rule.
When a rule that looked good in development fails in the holdout, stop calling it a validated thesis. The holdout has done its job. You can investigate why the result changed, but its dates are no longer an independent grade after you use them to alter the rules. Using a holdout period properly means preserving that boundary, including when it gives an answer you didn't want.
There is still useful work after a failed holdout. Read the qualifications. Check whether the thesis owned an unintended sector concentration or a few names accounted for the apparent development-period advantage. Then inspect its timing: filings must have been public before entry, and the exit assumptions must be practical. Those checks may explain the old result. They don't revive it.
The next experiment needs fresh evidence: a later period, a different but predeclared universe, or a claim narrow enough to stand on its own. Do not relabel the holdout as a discovery period and keep tuning until it works. Once data helps write the rules, it no longer grades them independently.
Keep a graveyard instead of a highlight reel
A thesis that fails should remain visible in your research record. Save the rule, the dates, the benchmark, the test settings, what you expected, and what happened. Write one sentence on the decision: retired because the result reversed under ordinary threshold changes; inconclusive because the historical sample was too sparse; revised because the original measure did not match the stated economic claim. You won't remember the details later. The record will.
That record prevents a costly loop. Months later, the same appealing hunch will return in new words. Without the old run, it feels untested. With the old run, you can see whether the new idea is genuinely different or just the previous thesis with a different parameter.
Quantery keeps a thesis version attached to each run, so the calculation that produced a result doesn't get overwritten by later edits. Use that history as a lab notebook. The rules are yours to inspect, change, and re-test. The failed versions are yours too. Deleting them makes the process look cleaner while making the next decision worse.
A research record also makes productive revisions possible. Suppose a rule failed because it was a sector bet. The next claim might be: within each sector, companies with stronger cash conversion fare better than peers. That changes the comparison the thesis makes. It earns a new test. Testing whether a screen is just a sector bet shows why that distinction matters.
The stop rule belongs in the thesis
Before your next backtest, write four lines:
- What claim does this rule make?
- What result would count against it?
- Which nearby settings will you test without changing the story?
- What evidence will remain unseen until the rule is frozen?
Then run the declared version, record the result, and let a failure stay failed. Even a robust historical result is a reason to investigate. It isn't a promise about the next period.
The useful outcome is a smaller set of rules that survived a process you can describe without embarrassment, plus a clear record of the rules that did not. That gives the next thesis a better starting point than another round of edits ever will.
Research tooling, not investment advice. Nothing here is a recommendation to buy, sell, or hold any security. Screens, scores, and backtests are informational only; backtested results are hypothetical, exclude costs such as commissions and slippage, and do not guarantee future results. Verify against primary filings and make your own decisions.
Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.
← All articles