← Quantery Blog

Does your backtest depend on one big winner?

September 26, 2026 · 9 min readbacktestingrobustnessoutliers

A backtest depends on one big winner when removing that company, event, or entry cohort changes the result's direction or erases most of its excess return. Check this by ranking contributions to the result, comparing the mean with the median and hit rate, then rerunning the test without the largest contributor. If the conclusion collapses, the broad thesis hasn't been established. You've found an outlier-dependent result.

Outlier dependence isn't automatically a defect. Individual stock returns are right-skewed: a few large gains can matter far more than the many ordinary outcomes. But a backtest carried by one exceptional path supports a narrower claim than a result repeated across companies and dates. Find out which one you have before you start explaining the curve.

Why can one winner own the result?

A stock can lose only the money committed to it, while its possible gain has no equivalent ceiling. That asymmetry makes the distribution of individual stock returns right-skewed, meaning the long tail stretches toward unusually large positive outcomes. Compounding makes the gap wider over long holding periods.

Hendrik Bessembinder's study of common stocks in the CRSP database found that the majority produced lifetime buy-and-hold returns below one-month Treasury bills. Measured in dollars, the best-performing 4% of listed companies explained the US stock market's net gain from 1926 onward; the others collectively matched Treasury bills (the paper's abstract and citation). That doesn't say a given backtest will be carried by one stock. It says the possibility belongs in your default checklist.

A screen can amplify the effect. A strict gate may admit few companies. A long holding rule gives an extreme winner time to compound. Repeated qualification can let one company appear in several event rows. An equal-weight portfolio can still become concentrated between rebalances as its largest winner grows.

This is why a high average return doesn't tell you whether the rule worked broadly. Breadth means the result appeared across enough independent companies and entry dates that one exceptional observation doesn't control the verdict. Breadth isn't the same as a large event count. Ten entries from the same company are one economic story wearing ten timestamps.

Read the distribution before the curve

Start with the event study. A backtest report separates qualification events from the portfolio policy, and that separation is useful here. Pick one forward horizon, then read the event count, mean excess return, median excess return, and hit rate together.

Excess return is a stock's return minus its benchmark's return over matching dates. The mean is the arithmetic average of those event outcomes. The median is the middle result after sorting them. The hit rate is the share that beat the benchmark.

A mean far above the median is the first warning. A small group of large positive events may be lifting the average while the typical event did much less. If the mean is positive but the median and hit rate sit below their neutral points, the screen may have found rare home runs among common misses. That can be a real profile. It just isn't evidence that the typical qualifier worked.

Don't demand a perfectly symmetric distribution. That would throw away a basic feature of stock returns. Quantery's go-event rows show the qualification date, symbol, and quantitative score. They can reveal repeated entries and date clusters. They don't show forward returns, so the report can't identify upper-tail events.

The portfolio curve shows when the account moved, but it doesn't expose per-holding weights or period returns. Attributing the move to one position requires a trade-level reconstruction outside Quantery. If several reconstructed contributors share a sector or entry window, use the sector-concentration test or a cohort removal next.

Calculate contribution outside the report

Contribution asks how much each holding added to the portfolio result under the actual weighting and holding policy. Don't rank stocks by raw return alone. A huge return in a tiny position may matter less than a moderate return held at full weight for months.

Quantery's current report doesn't provide the inputs for this calculation. It persists portfolio entry and exit positions and aggregate interval returns, but doesn't expose each holding's weight and return for each interval. Go events identify signals without describing the portfolio's allocation through time.

A reconstruction needs the original entry and exit dates, the weighting rule at every rebalance, security prices on those dates, and the treatment of cash and exits. Rebuild the original path before assigning contribution. For a simple interval, the arithmetic starts here:

contribution_i = starting_weight_i * holding_return_i

Across periods, link the path using the original rebalance dates and weights, then calculate a counterfactual path with the target holding removed. The difference in ending value is the holding's path-dependent contribution. If the reconstruction doesn't reproduce the report's aggregate path, stop. The attribution is answering a different question.

Decide the reassignment rule before calculating. If you remove a position, does its weight remain as cash, move pro rata to the other holdings, or trigger a fresh portfolio selection? Those are different tests:

• Cash replacement asks how much the omitted holding itself contributed.
• Pro-rata replacement asks whether the rest of the selected basket could carry the result.
• Fresh selection asks what the full policy would have done if that company had been ineligible.

Use one as the primary test and label it. Cash replacement is the cleanest attribution. Fresh selection is more realistic, but another company can enter and introduce a new effect.

Don't add contribution percentages from overlapping events. If one company qualified repeatedly while already held, those event-window returns share prices and time. Attribute the portfolio at the position level first.

Run a leave-one-winner-out test

A leave-one-winner-out test repeats the same hypothetical backtest after excluding the largest positive contributor. Keep the thesis rules, dates, benchmark, reporting lag, rebalance cadence, and exit policy fixed. Change one thing: the company can't enter.

Quantery doesn't currently offer a per-symbol exclusion or a contribution-ranked rerun. Do this in the external reconstruction that identified the contributor. First replay the original policy and confirm that its aggregate path agrees with the Quantery report. Then make the chosen company ineligible on every simulated date and replay the policy. Editing a present-day ticker list after the fact won't reproduce changing historical eligibility.

Read the pair in this order:

1. Does excess return keep the same sign?
2. How much does the gap versus the benchmark shrink?
3. Does maximum drawdown change?
4. Does another company become dominant?
5. Do the event-study median and hit rate still support the story?

If the sign reverses, the thesis result depended on that company in the tested sample. The rule admitted one extreme winner, while the remaining selections didn't beat the benchmark under this policy. Ask whether the rule had a defensible reason to identify that winner before the move.

If the sign survives but the magnitude falls hard, preserve the directional finding and stop quoting the original headline return as if it were portable. If removing each of the leading contributors leaves a similar direction, the evidence is broader.

Then remove the leading contributors one at a time, using an order declared before the reruns. Don't keep deleting winners until the test fails. Record the whole sequence.

Losers deserve a mirror test. A strategy can look dead because of one collapse just as it can look brilliant because of one runaway winner. Leave out the largest negative contributor after finishing the winner analysis. If either side controls the verdict, say so.

Every return in these reruns is hypothetical and excludes trading costs. Removing one company can change exposure and trading activity. The report doesn't calculate turnover, so your reconstruction must measure any change if you want to compare it. A replacement portfolio that trades more often hasn't held implementation constant.

Test where the dependence lives

Removing one ticker catches company dependence. It won't catch a result carried by several companies exposed to the same event.

Define an entry cohort as the companies entering at one rebalance date or within a predeclared calendar window. Rank cohorts by contribution and rerun without the leading cohort. If the result disappears, the thesis may depend on one market episode. That's especially important when a scan finds several companies from the same industry after a shared shock.

Next group events by sector. Several apparent winners can be one industry bet in disguise. Remove the dominant sector without changing the rules for everyone else. If that kills the result, narrow the claim to the sector and use a matched benchmark before testing again.

Finally, split the calendar using rules you could have known at the time. A regime test needs a reproducible definition, not a name attached after seeing the chart. Outlier dependence can migrate: one company may dominate the full period because one regime favored its industry.

These removals answer different questions. Company removal tests idiosyncratic dependence. Cohort removal tests timing dependence. Sector removal tests exposure dependence. Don't combine them into one giant exclusion and then wonder which diagnosis mattered.

What result is broad enough to keep testing?

There's no universal concentration cutoff that turns a fragile result into a sound one. A focused thesis will naturally have fewer independent bets than a broad quality thesis. Judge breadth against the claim.

If the claim says a common accounting pattern identifies stronger outcomes, you want the median, hit rate, and contribution spread to show repetition. If the claim says a rare setup occasionally finds extreme convex payoffs, a low hit rate and a strong upper tail may fit the mechanism. That second claim still needs enough independent episodes to separate a repeatable tail from one lucky draw.

Predeclare what would worry you. For example: a sign reversal when the leading company is removed, the leading cohort supplying most of the advantage, or the median staying below the benchmark across reasonable horizons. These are research conditions you're proposing, so choose them before looking at the contribution ranking.

Preserve the Quantery report as the baseline, then preserve the reconstruction code, input dates, exclusion list, and every counterfactual output beside it. Quantery stores the original thesis version and report. The contribution and exclusion results live outside the app, so reproducibility depends on keeping their assumptions attached. The broader robustness workflow applies next to the test window and its other assumptions.

A beautiful curve can be one company's biography. Contribution analysis tells you whose biography it is. Removal tests tell you whether the thesis has anything left when that story is gone. Keep the broad claim only when the result survives at that broader level. If it doesn't, narrow the claim and test again.

Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.

← All articles