← Quantery Blog

How to build a backtest sensitivity map

October 3, 2026 · 8 min readbacktestingrobustnesssensitivity-analysis

A backtest sensitivity map shows how the same thesis behaves across a small set of reasonable assumptions. Choose the settings before running them, change those settings systematically, and record the same outcome from every run. You’re looking for a region where the conclusion survives instead of the single cell with the highest return.

That last distinction is the job. A good-looking backtest can be one lucky address in a bad neighborhood. If nearby thresholds and test mechanics tell the same broad story, the result has earned another test. If one exact combination works and its neighbors fail, the map has found a tuning problem.

What does a sensitivity map test?

Every backtest contains choices that the market didn’t hand you. You chose the valuation cutoff. You chose the minimum quality score. You chose how often the rules would be evaluated. Even a round, defensible choice could have been a nearby round, defensible choice.

A sensitivity map is a grid of controlled reruns across those nearby choices. Each cell contains one consistently measured result, such as median excess return at a fixed event horizon. Excess return means the thesis return minus the matched benchmark return over the same interval.

The map asks a stricter question than “did this version work?” It asks whether the conclusion depends on a setting that your economic story didn’t make special.

Suppose your thesis rewards cash-generative companies at modest valuations. There’s no business reason why a free-cash-flow yield cutoff of 0.06 should reveal an edge while 0.05 and 0.07 reveal nothing. The companies around the boundary won’t become different businesses because one decimal place moved. A sharp result cliff at that line is evidence about the test, not evidence about the world.

This extends the one-dial perturbation in how to tell whether a backtest result is real. A perturbation compares a baseline with its neighbor. A map shows the neighborhood.

Which settings belong on the map?

Map the uncertain choices that could reasonably have been made before you saw the output. Start with one or two dimensions. More dimensions make the research harder to read and create more chances to admire a winner.

Good candidates include:

• A thesis threshold, such as a minimum yield or maximum leverage ratio.
• A score gate, tested at neighboring integer totals.
• A backtest mechanic, such as weekly versus monthly evaluation.
• A reporting lag, if your process could plausibly act at either setting.
• A holding rule, tested across nearby horizons that fit the thesis.

Keep choices with different meanings in separate maps when you can. A grid crossing a valuation threshold with a quality threshold says something about the thesis surface. A second grid crossing rebalance cadence with maximum hold says something about implementation. Mixing everything at once makes an attractive cell hard to diagnose.

Don’t map settings that you already know are invalid. A future filing date, a universe with known survivorship leakage, or an unrelated benchmark isn’t a robustness variant. It’s a broken test. Use negative controls to probe deliberately broken mechanisms and label them as such.

Also resist widening the ranges until a favorable patch appears. Define the neighborhood first. If the original parameter was a round belief about business economics, nearby values should be round too.

How do you design the grid before running it?

Write down the dimensions and values. Then define one outcome and a failure rule. Do this while you can still be disappointed by the result.

For example:

# Sensitivity-map plan: illustrative

thesis_dimension:
  parameter: fcf_yield_floor
  values: [0.04, 0.06, 0.08]

implementation_dimension:
  rebalance: [weekly, monthly]

fixed:
  benchmark: SPY
  event_horizon_months: 12
  universe: same_saved_definition
  reporting_lag_days: 1
  exit_rule: unchanged

primary_cell_value:
  statistic: median_excess_return

failure_rule:
  conclusion_fails_if: direction_depends_on_one_cell

The shape is illustrative. Those values are proposed research parameters, and the bundled templates are the reference for exact thesis fields. The Quantery DSL documentation covers the available structure.

This plan produces a compact set of runs. More important, it freezes what each cell means. The benchmark, date window, universe, reporting lag, exit rule, and event horizon stay fixed. Change two of those midway and you’re no longer filling the same map.

Choose one primary cell value. Median excess return is useful when a few large winners could pull the mean around. You might choose benchmark-relative portfolio return instead if your question is about the full holding policy. Either can work. Don’t fill one cell with whichever panel looked kinder.

Record supporting evidence beside the primary value: event count, hit rate, mean excess return, maximum drawdown, and coverage. These aren’t extra winners to shop among. They tell you whether two similar-looking cells were built from comparable evidence.

Every backtest result here is hypothetical and excludes costs. That caveat becomes more important when the map includes a faster evaluation schedule or shorter holding rule, because an improved gross result can arrive with more turnover.

What shapes should you look for?

Read the whole surface before reading its best cell. Several shapes recur.

A plateau is a connected patch where the conclusion keeps the same direction and the supporting evidence remains recognizable. The values don’t need to match. They won’t. A plateau says the thesis can tolerate reasonable disagreement about the exact setting.

A slope changes gradually as a rule tightens. That can fit an economic story. Demanding stronger cash generation may produce fewer events and a stronger typical outcome. Check that the improvement isn’t purchased entirely by a vanishing sample or one sector.

A cliff appears when neighboring settings reverse the result. Inspect the membership changes around that boundary. One company, filing cluster, or market episode may be carrying the difference. If there’s no economic reason for the boundary to be special, the result hasn’t earned special treatment.

An island is one attractive cell surrounded by weak or opposite results. Treat it as a failed robustness check. Don’t rename the exact settings as a newly discovered strategy. The island is what a search through many alternatives is expected to produce occasionally.

A ridge is a narrow line of good cells, often indicating that two parameters are compensating for each other. A looser quality rule may work only with a tighter valuation rule. That interaction can be real, but it’s now part of the claim and needs fresh evidence.

A flat map shows little relationship between the setting and the outcome. That may mean the parameter isn’t doing useful selection work, or that most observations sit far from its boundary. Compare survivor membership before deleting it. A rule can be economically sensible yet inactive over the tested sample.

The map’s geometry is descriptive. It doesn’t supply a p-value, prove causality, or turn correlated runs into independent evidence.

Why isn’t the best cell the answer?

Because the best cell won a contest you created. The larger the search, the less surprising it is that something looks exceptional.

Bailey, Borwein, López de Prado, and Zhu show that testing even a modest collection of strategy variants can produce an impressive in-sample result, while the number of discarded attempts is seldom visible to the reader. Harvey, Liu, and Zhu make the same point across published return factors: the usual significance hurdle is too low after extensive factor searching.

A sensitivity map doesn’t cure multiple testing. It makes the search visible. That’s useful only if you preserve every cell, including the awkward ones, and refuse to promote the maximum as the estimate.

The runs share data too. Neighboring cells often select many of the same companies on the same dates. Their outcomes are correlated. You haven’t created independent confirmation by running several close variants. Event-study research has the same warning: Kolari and Pynnönen find that positive cross-sectional correlation in abnormal returns can make conventional tests reject too often. A cluster of signals during one market episode may look like many observations while carrying much less independent information.

Use the map to judge stability, then spend untouched data on the final claim. A holdout period must come after the mapping work. Once the holdout result sends you back to change the grid, that period has become research data.

How should you run the map in Quantery?

Start from one saved thesis version and one baseline backtest. Duplicate the thesis for each threshold value, changing only the named parameter. Then run each saved version with the fixed backtest settings. If cadence is the second dimension, repeat the same thesis versions at the alternate cadence.

Name versions so the map can be reconstructed without memory. A plain convention works:

fcf04-weekly
fcf04-monthly
fcf06-weekly
fcf06-monthly
fcf08-weekly
fcf08-monthly

For each recorded report, copy the primary value and the supporting fields into your research note. Read the coverage note first. A result from a thin early universe isn’t comparable with a neighboring run that saw a broader set of eligible companies. Read the event count too, but remember that repeated qualifications can share a company or market episode.

Then inspect the cells that disagree. Compare the underlying scan runs to see which companies were added or dropped when a threshold moved. If cadence caused the change, inspect signal dates and boundary churn. If a longer hold caused it, check whether a small cohort stayed open long enough to dominate the portfolio.

Quantery keeps the thesis version and settings attached to each report. Use that record as the map’s audit trail. Don’t overwrite the baseline and rebuild the history from memory later.

What does a sensitivity map let you conclude?

A broad plateau lets you say the historical conclusion survived the nearby choices you declared. A cliff lets you say the conclusion depended on a narrow rule. An island tells you the winning configuration shouldn’t advance without fresh evidence.

None of those statements predicts what happens next. The map still covers one historical record, uses a chosen universe and benchmark, and excludes trading costs. It can expose fragility. It can’t certify a thesis.

Keep the map small enough to understand. Save every version. Advance a region, never a cell. Then freeze one representative set of rules and test it on data the map never touched. The point isn’t to find settings you can defend after the run. It’s to learn whether the idea survives when the settings remain yours to change.

Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.

← All articles