← Quantery Blog

What makes a backtest honest

August 6, 2026 · 6 min readbacktestingmethod

A backtest is a machine for answering one question: if I had followed these rules in the past, what would have happened? It's the best tool an individual investor has for separating ideas that worked from ideas that merely sound good. It is also, run carelessly, the best tool ever invented for lying to yourself with charts.

The lies aren't usually deliberate. They come from a handful of well-known defects in how historical data gets assembled and used — each one nudging results upward, each one invisible in the output. A 24% annualized backtest and a 6% annualized backtest look identical on screen; the difference is often just which of these defects went uncorrected. If you're going to trust any backtest — from us, from a paper, from your own tinkering — you should know all five.

1. Point-in-time data, or: no peeking at restatements

Companies restate. A 10-K filed in February gets amended in August; revenue recognition gets corrected; a segment gets reclassified. Most convenient data sources carry the final, cleaned-up numbers — which means a backtest running on them is trading, in simulated 2019, on figures that didn't exist until 2020.

That's not a small technicality. Restatements cluster around exactly the situations screens care about — deteriorating businesses, aggressive accounting, fraud — so the error is systematically flattering: your simulated self quietly avoids the blowups that fooled everyone in real time, because it saw the corrected numbers.

The fix is point-in-time data: for any simulated date, use only what had actually been filed by that date, as it was filed. This is why Quantery's fundamentals come from SEC EDGAR filings indexed by when they became public, not by the fiscal period they describe. Your backtest in March 2019 sees the world as a diligent filer-reader would have seen it that March — including the numbers that were later revealed to be wrong. That's the point.

2. Survivorship bias, or: the dead must vote

Take today's stock list, run your screen backward ten years, admire the returns. The defect: every company on today's list is, by construction, a company that made it to today. The ones that went bankrupt, delisted, or got acquired at a discount have been quietly removed from the ballot — and those are disproportionately the value traps your screen would actually have bought.

Cheap-stock screens are the worst hit, because "optically cheap" describes both future winners and future corpses, and a survivorship-biased universe deletes the corpses. Results on such a universe aren't slightly optimistic; entire published "strategies" have evaporated when re-run with the dead included.

Honest backtesting keeps delisted names in the historical universe, so your simulated screen can buy the losers it would really have bought. Quantery is survivorship-aware and built to be honest about its own limits here: from the day your install starts recording daily universe snapshots, the record going forward is leak-free by construction; history before your recording began is reconstructed on a best-effort basis, and results that lean on it are flagged as such rather than passed off as clean.

3. Look-ahead bias, or: knowing things early

Subtler than the first two: using information on a date when it technically existed but wasn't yet knowable. A fiscal year ends December 31; the 10-K arrives in late February. A backtest that trades January 2 on "last year's earnings" is using a document that won't be public for seven weeks. Rebalancing on the period date instead of the filing date is the single most common way homemade backtests cheat without meaning to.

The same discipline applies everywhere: prices as of the decision date, fundamentals as of the filing date, universe membership as of the moment. Quantery's engine enforces filing-date alignment for you — but knowing why matters when you read anyone else's results.

4. Costs, or: every backtest is hypothetical

Backtests, Quantery's included, generally exclude commissions, bid-ask spreads, slippage, borrowing costs, and taxes. For a patient low-turnover screen the drag may be modest; for anything trading small-caps or rebalancing frequently it can eat the entire apparent edge — spreads on thinly traded names are brutal, and screens love thinly traded names.

So treat every backtested return as an upper bound on a hypothesis, not a forecast of your account balance. This isn't a Quantery caveat; it's the nature of the exercise, and any tool that doesn't say so plainly is selling you something.

5. Walk-forward, or: don't grade your own homework

Tune a thesis until the backtest looks great and you have — a thesis tuned to that backtest. With enough dial-turning, any dataset will confess to anything. The honest structure is walk-forward: at each simulated date the rules act only on data available up to that date, then the clock advances and the results accrue on periods the tuning never saw. Combined with discipline about when you stop tweaking, it's the difference between "this rule had an edge" and "this rule memorized the past."

Quantery's backtests run walk-forward, pair an event study (what happened around each qualification) with a naive portfolio, and benchmark each run against SPY or IWM so "it went up" gets compared to "everything went up." And because backtests are unlimited — no metering, no per-run fees — you can afford the honest workflow: many runs, varied assumptions, checking whether results survive perturbation instead of cherry-picking one beautiful curve.

What a good result actually tells you

Suppose you do everything right — point-in-time fundamentals, survivors and corpses alike, filing-date alignment, walk-forward — and the result is still good. What do you now know?

You know that these rules, applied with this discipline, over this period, would have beaten this benchmark before costs. That is a real and valuable fact. It kills bad ideas by the dozen, and it earns good ideas the right to real money — usually small money first.

Here's what it does not tell you:

• That the future resembles the period tested. Every backtest is a sample from one historical draw. Regimes end; edges get crowded and arbitraged away.
• That you would have held on. The equity curve compresses years into pixels. Somewhere in that curve is an eighteen-month stretch of underperforming while your neighbor's index fund wins. The backtest assumes you kept following the rules through it. Would you?
• That the result is causal. A screen can win for reasons that have nothing to do with your story about it — a sector tilt, a size tilt, one lucky cohort. This is why the event-study view and per-name drill-down matter: look at what it bought, not just how the line ended.
• That anyone should act on it. A backtest is research. It is not advice, ours or anyone's.

The practical checklist

Reading anyone's backtest — including your own — five questions: Point-in-time fundamentals, or restated? Are delisted names in the universe? Trades on filing dates, or period dates? What costs are excluded? Was it tuned on the same data that graded it? Anyone publishing results should be able to answer all five without flinching. For Quantery's engine the answers are: point-in-time from SEC filings, with fundamentals supporting backtests from 2013 (earlier concept gaps are flagged, and price depth depends on your market-data plan); delisted names retained, with reconstructed pre-snapshot history flagged; filing-date aligned; costs excluded and labeled hypothetical; walk-forward by construction.

None of this rigor makes a backtest predictive. It makes it honest — a fair record of what your rules would have done. That's the most any backtest can be, and it's enough: it's the difference between testing your ideas and decorating them.

Want to try this on your own rules? Quantery is free for 14 days — the full app, no card required.

← All articles