QUANTERY

← Quantery Blog

How to keep a backtest research log

October 6, 2026 · 9 min readbacktestingresearch-logmethod

A backtest research log is a record of the question you asked, the version of the rules you tested, the result you got, and the decision you made before moving on. Keep one row per run. Write the expected failure condition before you see the report, then record failures as carefully as successes.

That sounds like paperwork. It isn't. The log is what separates a reproducible research process from a folder full of attractive charts. Without it, you can't tell whether today's promising result was your original idea or the winner of a long, mostly forgotten search. You also can't reproduce the path that led there.

Why does a backtest need a research log?

Backtesting invites selective memory. You change a threshold, move the start date, try a different benchmark, and alter the rebalance cadence. By the end of the afternoon, you remember the best curve and perhaps the version that made it. The dead ends blur together.

Those dead ends matter because every tried specification is part of the evidence. If many plausible variants failed and one narrow variant worked, that lone success deserves more suspicion than it would in isolation. The search was larger than the final report admits.

This is the practical version of multiple testing: the more variations you try, the more likely chance is to hand you an impressive outcome. Bailey and coauthors describe how investment strategies can be overfit when researchers search many configurations and retain the best performer in their paper on the probability of backtest overfitting. Harvey, Liu, and Zhu make the related point across the published factor literature: ordinary significance thresholds become too forgiving when the number of tried explanations is large (their survey of the factor search).

A personal log won't solve the statistical problem. It does stop the search from disappearing. That's enough to change how you read the winner.

The log also handles a more mundane failure: forgetting what changed. Two reports can differ because of the thesis rules, the eligible universe, the date window, or the benchmark. If several inputs moved together, the comparison teaches you almost nothing. A run record makes that visible while you can still fix it.

What belongs in each run record?

Write enough to recreate the question. You don't need every thought you had while waiting. A useful record has these fields:

The fields force a distinction between observations and decisions. "The result weakened when the threshold moved" is an observation. "Reject this threshold-specific claim" is a decision. Keep both. Otherwise it's easy to explain away the result after seeing it.

Don't turn the log into an essay. A plain text file, spreadsheet, or notebook works. Quantery already preserves thesis versions and backtest reports, so your note can point to those records instead of copying every output. The valuable part is the intent that existed before the run and the decision made afterward.

Write the failure condition before running

A research log is most useful in the few minutes before a backtest. Write a sentence that gives the result permission to disappoint you.

Suppose the thesis claims that companies with improving cash conversion tend to outperform a matched benchmark. Before the first run, you might write:

I will weaken the claim if its excess return changes sign when the cash-conversion threshold moves across nearby round values, or if the result appears only in one date window.

This isn't a universal rule. It's a commitment for this test. It prevents a familiar move: changing the meaning of success after the report arrives.

Do the same for a successful outcome. Define what it would earn. One good full-period result might earn a threshold perturbation and a holdout test. It shouldn't earn confidence by itself. A holdout period is data deliberately set aside while you develop the thesis, then opened after the rules are frozen. If you keep editing after seeing it, it stops being a holdout and becomes another development window.

Your log should mark that boundary in advance. Record the development dates, the untouched test dates, and the condition under which you'll open the test. If the result fails there, keep the failure. Don't rename the test period "unusual" and return it to the workshop.

Change one thing per comparison

A clean comparison has a baseline and a variant with one meaningful difference. If you tighten the quality threshold, switch from a broad benchmark to a value benchmark, and shorten the holding period in the same run, a changed result has three possible explanations.

Use a short sequence instead:

  1. Freeze the baseline and record its thesis version and settings.
  2. Change one parameter or one backtest assumption.
  3. Predict which part of the report should move and why.
  4. Run the variant and record the full result, including an unchanged result.
  5. Decide whether the next run follows the original plan or investigates a surprise.

An unchanged result can be informative. Perhaps the threshold never affected which companies qualified. Perhaps monthly rebalancing swallowed a small reporting-lag change. Check the events before declaring the thesis robust. A setting that didn't reach the selections wasn't really tested.

This one-change discipline is the working habit behind testing whether a backtest result is robust. The log makes the sequence legible after the fact. It also exposes when the sequence has become a hunt. If the "next run" column keeps changing in response to whatever looked good, stop and write a new research question.

Log failed runs without rewriting history

Failed runs are easy to lose because they don't produce a chart anyone wants to keep. Record them anyway. A failure can mean the mechanism was wrong, the proxy was poor, coverage was thin, or the test couldn't distinguish the claim. Those are different outcomes.

Use blunt labels:

Never delete a run because its settings were mistaken. Mark the mistake. A filing-date bug or mismatched benchmark is part of the audit trail, and retaining it prevents you from citing that report months later.

Coverage deserves its own line. The same rules can appear to change when the effective historical universe changes. Record how much of the intended universe the run could actually evaluate and copy any reconstruction warning from the report. Checking data coverage before a backtest explains the preflight; the log records what the completed run actually saw.

How should you log exploratory work?

Not every run begins with a pristine hypothesis. Exploration is how you learn what a dataset contains and whether an idea can be expressed at all. Label it.

An exploratory run can answer questions such as whether a feature has usable coverage or whether a gate leaves enough events to study. It can suggest a better proxy. What it can't do is serve as untouched confirmation of the pattern that suggested the proxy.

When exploration produces a promising rule, draw a line in the log. Freeze the rule, state the new claim, and test it on a period or condition that didn't choose it. This is where the distinction between development evidence and test evidence becomes useful. Both belong in the record, but they don't carry the same weight.

Don't clean up the chronology for presentation. If the current thesis came from several failed proxies and threshold changes, say so in your own record. The final specification may still be worth testing. Its origin determines how demanding the next test should be.

Let saved versions carry the rules

A Quantery thesis is saved as an immutable version, and each run records the version that produced it. Use that machinery as the spine of the log. Don't paste a loose copy of the rules into a note and assume it still matches the app.

A concise entry might read like this:

Question: Does the result survive a looser quality threshold?
Baseline: thesis v7, run bt_previous
Variant: thesis v8, run bt_current
Changed: quality_floor from 0.20 to 0.15
Fixed: universe, dates, benchmark, monthly rebalance, exit rule
Prediction: selections broaden; excess-return direction remains
Failure: excess-return direction reverses or coverage changes materially
Result: [complete after reading the persisted report]
Decision: [keep / reject / investigate]
Next: [write before launching another run]

The values here are proposed parameters. They don't describe a completed run. Replace the bracketed lines only from the persisted report. Include the run's hypothetical, before-cost framing in any summary you later share.

Use the report identifier as the durable reference. A screenshot helps communicate the result, while identifiers and versions let you reproduce it. The guide to reading a backtest report covers which output belongs in the result field.

When is the research log complete?

It isn't complete when the curve looks good. It's complete for a research question when the predeclared sequence has ended and the record says what you now believe.

Close with a short finding:

Then stop. A stopping rule belongs in the log because unlimited reruns make it easy to keep searching after the planned test has spoken. More computation doesn't rescue a rejected claim if each new run was chosen because the prior one disappointed you.

Quantery gives you editable rules, immutable thesis versions, and persisted reports. The log joins them into an argument you can inspect later. Keep the failures. Keep the abandoned branches. When a result survives, you'll know what it survived, and you can change the rules again without pretending the earlier search never happened.

Research tooling, not investment advice. Nothing here is a recommendation to buy, sell, or hold any security. Screens, scores, and backtests are informational only; backtested results are hypothetical, exclude costs such as commissions and slippage, and do not guarantee future results. Verify against primary filings and make your own decisions.

Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.

← All articles