← Quantery Blog

What does a backtest confidence interval mean?

October 17, 2026 · 8 min readbacktestingstatisticstesting-method

A backtest confidence interval is a range around an estimated result, such as mean excess return, produced by a stated sampling model. It asks how much that estimate might move if we could repeat the historical sampling process. It does not certify the thesis, predict its future return, or repair bad data.

That phrase, stated sampling model, carries the whole burden. A standard interval assumes the observations behave like separate draws. Stock-screen events often don't. Several companies can qualify on the same date, the same company can qualify again, and forward-return windows can cover the same market move. Ignore those links and the interval can look much tighter than the evidence deserves.

What question does a confidence interval answer?

Start with an estimand, the exact quantity you want to estimate. In an event study it might be the mean benchmark-relative return after a new qualification. The backtest supplies a sample estimate of that quantity. A confidence interval adds a measure of sampling uncertainty around the estimate.

For a mean, the familiar shape is:

interval = sample mean ± critical value × standard error

The standard error describes how much the estimated mean would vary across repeated samples under the model. The critical value sets the chosen coverage level. The NIST explanation of confidence limits for a mean makes the interpretation precise: a 95% procedure is designed so that, over repeated samples, about 95% of the intervals produced contain the population mean. It isn't a claim that this particular completed interval has a 95% chance of containing a fixed truth.

That distinction matters because a backtest gives you one historical path. You don't get repeated capital markets with fresh draws. The interval is therefore conditional on a model for how the observations could have varied. Change that model and the range changes.

An interval also belongs to one statistic. An interval for mean excess return says nothing direct about the median event, the share of events that beat the benchmark, maximum drawdown, or the ending portfolio value. Those are separate estimands with different sampling behavior. Pick the quantity that matches the thesis claim before reaching for a formula.

Why is the simple interval usually too confident?

The textbook mean interval gets narrower as the sample grows because its standard error divides dispersion by the square root of the observation count. That arithmetic is useful when each observation contributes fresh information. It breaks when many rows repeat the same economic episode.

Picture a value thesis that admits a group of industrial companies at one rebalance. Their subsequent returns share rates, input costs, and the market path. Treating every company as an independent draw lets one episode vote many times. A later re-entry by the same company can add another related row. Forward windows beginning near each other may overlap as well.

This dependence has two forms. Cross-sectional dependence links different companies exposed to a common event. Serial dependence links outcomes across time. NIST's autocorrelation definition describes the latter as correlation between observations separated by a lag; material autocorrelation means the randomness assumption has failed.

The event-study problem is well documented. Kolari and Pynnönen found that when event dates cluster, even modest cross-correlation in abnormal returns can make conventional tests reject a no-effect hypothesis too often (The Review of Financial Studies). A narrow interval calculated from every event row can be mathematically correct for an independence model that the backtest plainly violates.

Read clustered backtest signals before reading the interval. Count distinct qualification dates, recurring companies, and shared sectors. The raw event count describes rows. It doesn't tell you how many separate chances the thesis had to fail.

Which observations should be resampled together?

A practical answer is to keep related observations together. If companies qualify on the same rebalance date, treat that date cohort as a block. When several nearby dates share overlapping forward windows, use calendar blocks long enough to preserve that overlap. Draw blocks with replacement, recompute the statistic for each resampled history, and take the chosen quantiles of those resampled estimates.

The outline is simple:

for each resampled history:
  draw calendar blocks with replacement
  keep every event inside each selected block
  recompute the chosen statistic

interval = lower and upper quantiles of the resampled statistics

This is a block bootstrap. Unlike resampling isolated rows, it preserves some of the dependence inside each block. Politis and Romano's stationary bootstrap is one established version for weakly dependent stationary observations. Its existence doesn't settle your block design. Market episodes don't arrive with labels, and a financial return series can change character across the sample.

So vary the block definition. Try date cohorts, then longer calendar blocks that preserve nearby rebalances. If the conclusion changes when the block length changes, keep the wider uncertainty in view. A block chosen after inspecting which version gives the neatest answer is just another tuned parameter.

For a thesis that wakes up during rare industry shocks, the episode may be the proper unit. Resampling companies inside one shock creates the appearance of many histories when you still have one shock. This is the same lesson as counting independent go events: match the unit of evidence to the width of the claim.

How do you build the interval without fooling yourself?

Freeze the research decision first. Write down the thesis version, benchmark, return horizon, entry convention, exit rule, and statistic. Then preserve every event needed to reproduce the reported point estimate. If your external calculation can't match the report, stop. An interval around a different estimate answers a different question.

Next, inspect the event anatomy before choosing the method:

• Group new qualifications by rebalance date.
• Mark repeat qualifications by the same company.
• Mark overlapping forward windows.
• Check whether large cohorts share an industry or another economic exposure.
• Record missing outcomes and explain why they are missing.

Choose the resampling unit from that inspection, then declare it in the research log. Recompute the entire statistic within every draw. Don't bootstrap an already averaged table, because that table may have erased the cohort structure you needed to preserve.

Report the point estimate and interval beside the sample anatomy. Include the number of event rows, distinct companies, distinct date cohorts, and the block rule. Those counts need to come from the saved run or your reproducible event table, rather than memory. A reader should be able to tell whether the range rests on a long stream of separate decisions or a few crowded periods.

Quantery's saved backtest report is the baseline for this work. It preserves the thesis version and settings, and it reports descriptive horizon summaries. It doesn't currently calculate statistical confidence intervals or expose event-level forward returns. Building one therefore requires a separate event-return table that reproduces the report before resampling begins. Keep that table, the code, and the block choice beside the persisted run. The backtest research-log workflow is a good place for all three.

What can no confidence interval fix?

A confidence interval handles sampling uncertainty under its assumptions. It can't rescue look-ahead bias, survivor-only universes, stale filings, a mismatched benchmark, or missing trading costs. A precise interval around a contaminated estimate is precise contamination. Start with the data checks in what makes a backtest honest, then ask how uncertain the remaining estimate is.

It also can't erase the search that selected the winning thesis. If you tried many thresholds, universes, and holding rules, then wrapped an interval around the best result, the interval conditions on the winner while ignoring the discarded attempts. Harvey, Liu, and Zhu show why conventional significance cutoffs become too permissive when many factors have been tested (NBER). Bailey and coauthors make the same problem concrete for strategy research: impressive simulated performance can emerge after trying a modest set of configurations, especially when the size of that search goes undisclosed (Western Michigan University ScholarWorks).

Keep a holdout period untouched while designing the thesis. Freeze the rules, run the holdout once, and calculate its uncertainty using the dependence structure you declared beforehand. The interval still won't promise that the next regime resembles the sample. It will answer a cleaner question because the same observations didn't choose the rule and grade it.

How should you read the finished range?

If the point estimate is positive while the interval spans outcomes on both sides of zero, the sample hasn't pinned down the direction under your model. If the whole range is positive but widens sharply under cohort or calendar-block resampling, dependence was carrying some of the apparent precision. If reasonable block choices preserve the direction and produce similar ranges, the estimate has survived a useful challenge.

None of those readings turns the interval into a verdict. Read it beside the median, event distribution, portfolio path, coverage record, and robustness runs. Backtested returns remain hypothetical and exclude costs. A range can describe uncertainty in the recorded history; it can't turn that history into a forecast.

The useful result is often less tidy than a single return figure: an estimate, a range, and a clear account of what was allowed to vary. Keep those together. Then change the rules only when the evidence teaches you something, and rerun the whole uncertainty calculation with the new version.

Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.

← All articles