← Quantery Blog

How many signals does a backtest need?

September 28, 2026 · 8 min readbacktestingmethodresearch

A backtest needs enough independent go events to give its claim a fair chance to fail. There isn't a magic minimum. The useful question is whether the result comes from many separate decisions, made across different dates and businesses, or from a handful of crowded cohorts that happened to work.

A go event occurs when a company newly passes a thesis's gate (its must-pass rules) on a rebalance date (one of the scheduled dates when the backtest re-evaluates the thesis). A company that remains above the gate at the next rebalance doesn't create another event. If it drops out and later passes again, that later transition does. Go events are the unit of evidence in a signal test. Count them, inspect when they formed, and ask how much of the conclusion survives when the most important group is removed. Do that before treating a return figure as a property of the rule.

This sounds like a statistical concern, and it is. It's also a design concern. A thesis that produces one go event every few years is making a different claim from one that produces a changing set of candidates every month. Neither is automatically better. They need different standards of interpretation.

Why does event count matter more than a smooth curve?

A portfolio curve compresses a lot of decisions into one line. It doesn't tell you whether that line came from a broad run of go events or one extraordinary stretch. If a thesis produced only a few events, an acquisition or bankruptcy can dominate the outcome. An extreme rerating can do the same. The end result may look decisive because the graph has no way to show how little evidence supports it.

Start with the event list. For every go event, record the date, the company, the thesis version, the score, the planned holding rule, and what happened afterward. Then group those records by formation date. A cohort is the set of companies that newly passed around the same rebalance. Cohorts tell you whether the thesis kept finding opportunities as conditions changed.

A thesis with go events spread through strong and weak markets has faced more varied evidence than one that happened to fire during a single burst of enthusiasm. Long dull stretches belong in the record too. That doesn't prove it'll travel into the future. It tells you whether the historical result rests on more than one local story.

Independence matters too. Ten companies qualifying on the same day can still amount to one economic bet if they share a sector, funding source, customer base, or reaction to the same news. Read the sector concentration and the individual events beside the count. Testing whether a thesis is a sector bet handles that separate question. Event count tells you how much evidence you have. Composition tells you whether that evidence is all the same evidence in different tickers.

What makes two signals independent enough to count separately?

Two events are more independent when the thesis made separate decisions with different information sets and different economic exposures. A company qualifying in one reporting cycle and another qualifying much later give the rule two chances to be wrong for different reasons. A cluster of companies passing after the same industry shock may still be useful evidence. Call it a cluster.

There is no clean switch from dependent to independent. Think in degrees:

• Same company dropping out and returning at the next rebalance: a new go event, but one heavily related to the first. The fundamentals may be from the same filing, and the price path overlaps.
• Different companies in the same industry on the same date: related through common demand and costs. Valuation pressure can link them as well.
• Different companies from different industries in separate reporting cycles: less related, though broad market conditions can still link them.
• A thesis that produces repeated go events for one company: evidence about recurrence instead of fresh independent discoveries.

Don't throw away the related events. Label them. Consecutive rebalances above the gate produce one go event, so don't count continued qualification again. If a company drops below the gate and later crosses back above it, the new event may teach you about recurrence. It still isn't the same evidence as finding a new case.

This is why a reported average return can mislead. Averages give every row equal weight even when several rows describe nearly the same underlying episode. Read the median and the distribution, then open the event list before leaning on one summary. How to read a backtest report explains what those summaries can show. The count beside them is context, not decoration.

When is a sparse result still worth testing?

A sparse result can be exactly what the thesis predicts. A deep-discount rule may be designed to wait. The same goes for rare corporate events or strict accounting gates. Loosening them until they produce a busier table changes the claim. Don't do that just to make a report look more comfortable.

Instead, state the finding at the scale the evidence supports. A sparse run can say: these precise conditions produced this small set of hypothetical historical events. It can't safely support a sweeping conclusion about every company with a similar-looking ratio.

There are several good next tests for a thin result:

1. Extend the test period without changing the rule. More history may add events, but only if the underlying data coverage supports the earlier dates. Don't attach a longer price history to fundamentals that weren't available at the time.
2. Use a nearby threshold defined in advance. A durable idea shouldn't depend entirely on one knife-edge cutoff. Move a round parameter modestly in each direction, preserve every run, and see whether the event stream disappears or changes shape.
3. Test the holding rule separately. A sparse thesis can look strong because one event was held through an unusual move. Stress-testing the exit rule shows how to separate selection from the decision to keep holding.
4. Treat a holdout period as a new trial. Develop the rule on an earlier segment, freeze it, and examine later events without further tuning. A quiet holdout is still information. A holdout-period workflow keeps the distinction clear.

The wrong response is adding conditions until a rare event becomes a perfect historical story. Every extra condition narrows the pool and makes it easier to explain a handful of surviving names after the fact. Keep the original rule visible. If a revised version is worth trying, save it as a distinct thesis version and make it earn its own record.

Know what an empty event stretch can tell you

Dates with no go events don't explain themselves. A company may still be above the gate from an earlier date, so no new event doesn't mean an empty candidate set. The backtest report records new gate crossings. It doesn't preserve the failed evaluations or historical near-misses needed to diagnose why other companies missed.

If that distinction matters to your claim, you need a separate record of each rebalance's full evaluation, kept outside the Quantery backtest report. It has to hold the near-misses and their criterion flags at each date. Then you can tell whether an economic criterion blocked candidates or a required field was missing. You can't reconstruct that history from the backtest report afterward.

The rule's construction still tells you what an empty stretch might mean. A relative rank will usually keep selecting some names as long as valid inputs exist. An absolute threshold can legitimately return none when the market offers nothing that meets the stated bar. But deciding which explanation applied requires those separately captured evaluations.

The shape below is illustrative. The bundled templates are the reference for exact fields and options. This example keeps the threshold named and separates data availability from an economic pass:

params:
  fcf_yield_floor: 0.05
  min_dollar_volume: 2000000

features:
  fcf_ttm: ttm(free_cash_flow)
  fcf_yield: if(market_cap > 0, fcf_ttm / market_cap, null)
  dollar_volume: avg_volume * price

criteria:
  cash_return:
    rules:
      - { when: "is_null(fcf_yield)", score: 0, flag: no_fcf_yield }
      - { when: "fcf_yield >= $fcf_yield_floor", score: 1 }
      - { else: 0 }
  tradeable:
    rules:
      - { when: "is_null(dollar_volume)", score: 0, flag: no_volume }
      - { when: "dollar_volume >= $min_dollar_volume", score: 1 }
      - { else: 0 }

gate:
  mode: strict

In a separately captured evaluation history, cash_return and no_fcf_yield answer different questions. The first says the rule saw a cash-yield result that missed its threshold. The second says the yield couldn't be calculated. Keep that record outside the backtest report if the distinction matters to your conclusion.

How should you report a thin backtest?

Report the shape of the evidence along with the result. Give the number of go events and their spread across time. Identify recurring symbols, sector concentration, and the handful of events with the largest influence. Do this for the baseline before you change a parameter. You want a record of what moved when the result changes.

Don't use the event count as a pass-fail badge. A long record of related events can still leave the claim narrow. A short record of well-separated events can justify more research while remaining too thin for confidence. The conclusion should match the evidence you have. It shouldn't match the evidence you wish the thesis had produced.

A count of independent-looking events doesn't turn a historical record into a forecast. It stops a single dramatic outcome from standing in for a method.

Run the thesis, open the events, and count the new gate crossings the rule actually produced. Then make the next change because that count and its composition taught you something. The rules are yours. The burden to show what they have really been tested against is yours too.

Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.

← All articles