← Quantery Blog

How to compare return horizons in a backtest | Quantery Blog

October 22, 2026 · 8 min readbacktestingevent-studytesting-method

Compare backtest return horizons as a sequence of separate measurements. For each horizon, read the sample size and hit rate. Compare mean excess return with the median. Then check whether later horizons use the same events and whether their return windows overlap. A stronger long-horizon average doesn't prove that waiting longer improved the thesis.

A return horizon is the interval between a fresh qualification and the date when its outcome is measured. Short horizons ask whether the market reacted soon after the signal. Longer horizons ask whether the result persisted or reversed. They may also capture gains that accumulated after the original signal grew stale. The useful object is the whole path across horizons, with its changing evidence attached.

Why does the sample shrink at longer horizons?

A new qualification needs enough post-event benchmark and security price coverage to produce an outcome. The chosen backtest end date limits when Quantery generates signals, but it doesn't cap outcome measurement: horizon dates come from the benchmark's full stored trading calendar. Stale or delisted outcomes don't disappear: Quantery carries the last available security price forward and applies the run's declared delisting haircut when that price is stale. An observation drops out only when a required starting price, horizon date, ending price, or benchmark price is unavailable. So the n beside each horizon belongs in the result.

Start with the shortest horizon and follow which observations remain eligible. If the long-horizon row looks much better while its sample has thinned, don't describe the difference as improvement until you've compared the same events at both horizons. The apparent gain may come from losing cases without benchmark or security price coverage through the later target date.

Missing outcomes can be informative. A company disappearing after distress isn't equivalent to a randomly missing price. A sound backtest needs a declared treatment for delistings, because dropping those rows can flatter the surviving sample. How a backtest should handle delisted stocks covers the terminal-return problem in detail.

Use a constant cohort check. Take only events that have valid outcomes at every horizon you plan to compare, then recompute the horizon summaries on that fixed group. This gives up some observations, but it answers the clean question: how did the same signals develop over time?

Keep the ordinary all-available rows too. They use more evidence at each horizon. The disagreement between the two views is useful. If the all-available path strengthens while the constant-cohort path doesn't, changing sample composition deserves more credit than signal persistence.

Read mean, median, and hit rate together

Excess return is a company's return minus the benchmark's return over the same interval. The mean is the arithmetic average of those excess returns. Quantery sorts the outcomes and reports the middle one, choosing the upper of the two middle observations when the sample size is even. The hit rate is the share of events that beat the benchmark.

Read them as a bundle:

• Rising mean with a flat median can mean a few large winners kept running.
• Rising median with a modest mean says the typical event improved while laggards held down the average.
• A falling hit rate beside a rising mean means fewer events won, but the winners became larger.
• A stronger hit rate with weak mean excess return says wins were common but small, or losses were fewer and severe.

None of these patterns is automatically better. They describe different distributions. A thesis claiming broad re-rating should produce evidence in the median and hit rate alongside the mean. A thesis built around rare asymmetric outcomes may expect the mean to lead, but then dependence on a handful of names is part of the claim.

Inspect the event list before assigning a story. One large winner can influence several horizons, so a smooth rise in the mean may be the same company voting repeatedly along its path. Testing whether a backtest depends on one big winner shows how to remove leading contributors while leaving the thesis fixed.

The benchmark must also remain aligned. Compute excess return against the same benchmark over exactly the same dates as each event. A short horizon measured against one market interval and a long horizon measured against another can't borrow a benchmark return from a headline elsewhere. Choosing a backtest benchmark starts with the alternative the screen could plausibly have replaced.

Why aren't horizon rows independent tests?

Return horizons from one signal are nested. The later outcome contains much of the earlier path. If a company is measured from qualification through successive dates, those rows aren't fresh trials of the thesis. Each row is another snapshot of one unfolding outcome.

Events can overlap across companies too. Several names may qualify on one rebalance date, or nearby signals may travel through the same market rally. This creates dependence across observations and across horizon rows. The NIST definition of autocorrelation explains that observations related across time violate the randomness assumption used by many standard statistical calculations. Event clustering adds cross-company dependence. Kolari and Pynnönen found that even modest cross-correlation can make conventional event-study tests reject the no-effect hypothesis too often when event dates cluster (The Review of Financial Studies).

That doesn't make the horizon table useless. It changes how much authority each row gets. Treat it as a descriptive path before treating it as a collection of significance tests. Count distinct dates, companies, and market episodes beside the event rows. A dense group of signals during one recovery is evidence about that recovery first.

If you calculate uncertainty outside the app, preserve the dependence. Resampling isolated event rows assumes away the very overlap you're trying to measure. Calendar-block or date-cohort resampling keeps related observations together. Backtest confidence intervals need an explicit sampling model, and the block choice should be declared before you inspect which version gives the tightest range.

Does a stronger long horizon mean the signal lasts longer?

Sometimes. A steadily improving median on a constant cohort, spread across several qualification periods, is consistent with a slow mechanism. A filing signal may take time to be noticed. An operational improvement may need another report before the market gives it credit.

But cumulative returns have more time to move. Their dispersion usually grows with the horizon. Broad market exposures have longer to matter too. A value screen held for a long interval can become a test of market exposure as much as a test of the filing that triggered entry.

Separate three possible shapes:

Fast response. Excess returns appear early, then flatten or reverse. The signal may be brief, or the initial move may have borrowed from the later interval.

Slow response. The median and hit rate strengthen over later horizons for the same cohort. This is consistent with gradual recognition, provided several dates contribute.

Late rescue. Early results are weak, while a distant mean turns positive because a few winners compound. This can still matter, but it doesn't support a broad quick-reaction story.

Don't pick the best horizon after reading the table and install it as the exit rule. That is tuning. A horizon analysis measures outcomes from an entry event. A maximum holding period governs a portfolio policy. They answer related questions, but the portfolio also contains overlapping positions, rebalances, and gate-fail exits. Stress-testing the exit rule keeps that policy question separate.

How should you run the horizon comparison?

Freeze the thesis version and universe. Keep the benchmark, reporting lag, entry convention, and test dates fixed too. Decide which mechanism you expect before opening the output. Quantery reports four built-in event-study horizons: 1m, 3m, 6m, and 12m. Then use a compact workflow:

1. Read the built-in horizons as one path from 1m through 12m.
2. Record n, mean excess return, median excess return, and hit rate at each one.
3. Recompute the path for the constant cohort with outcomes available throughout.
4. Group events by qualification date and mark repeat entrants.
5. Remove the largest contributor once as a diagnostic. Keep the original result as the baseline.
6. Compare the horizon path across fixed subperiods or an untouched holdout.

Those four endpoints are enough for the first comparison. If you need a different interval for a stated economic reason, calculate it outside Quantery from an event-level research table. Treat that as a separate analysis, not a backtest run setting. Adding many nearby endpoints after seeing where the line peaks creates another parameter search. The same warning applies to changing the benchmark or eligibility window until one row looks clean.

Quantery's event study supplies the four horizon summaries attached to a saved thesis version. Read it before the naive portfolio, as the backtest-report walkthrough explains. For external calculations, keep an event-level research table that reproduces the saved summary first. If your table can't match the report, don't use it to make a more elaborate claim.

All backtest returns are hypothetical and exclude trading costs. Longer holding periods may trade less often, but that fact can't be inferred from an event-study row. The event study follows qualifications. It isn't a transaction ledger.

What conclusion has the horizon table earned?

A defensible conclusion names the shape and its limits. For example: the same qualifying cohort showed broader benchmark-relative improvement at later measurements, but much of the evidence came from one qualification period. Or: the short-horizon mean was positive while the median wasn't, and the advantage disappeared after the largest contributor was removed.

Those sentences are less satisfying than “the signal works best if held longer.” They're also closer to what the test measured.

Look for a path across the four built-in horizons that survives the fixed-cohort test and appears across multiple dates. Check the median, then remove a large outlier once. Then test the implied holding policy separately. If an economic mechanism calls for another endpoint, calculate it outside Quantery and label it as external work. Each extra endpoint asks the historical sample another question, so keep the same events in view and make the conclusion no smoother than the underlying record.

Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.

← All articles