← Quantery Blog

How to test whether two stock signals work together

September 27, 2026 · 9 min readthesis-buildingbacktestingsignal-interactions

Two stock signals work together only when the effect of one changes in the presence of the other. Test that claim with four otherwise identical backtests: neither signal, each signal by itself, and both signals together. Then compare the same outcome across all four runs. If the joint version looks good only because it admits fewer companies or shifts into a favored sector, the combination hasn't earned the story attached to it.

This is a small factorial test, meaning every setting of one rule is paired with every setting of the other. The NIST handbook describes a full factorial design as running every high-and-low combination of the input factors. With two rules switched on or off, that gives four versions. A market backtest isn't a randomized laboratory experiment, but the structure forces a useful question: does the second rule add the same thing everywhere, or does it become more useful when the first rule is active?

What does it mean for signals to interact?

Suppose a thesis favors companies with a high free-cash-flow yield. Call that the value signal. You add cash conversion, free cash flow divided by net income, because cheap reported earnings are less attractive when the cash never arrives. Call that the cash-backing signal.

There are at least three possible findings.

• Independent contribution: the cash-backing rule improves the result by about the same amount with or without the value rule.
• Complementary contribution: cash backing adds more when the value rule is already active. The pair separates a useful subset that neither rule isolates alone.
• Redundant contribution: the second rule adds little because both signals react to much the same underlying fact or select nearly the same companies.

An interaction is the second case in its strict sense: the contribution from one signal depends on whether the other signal is present. It isn't enough for the combined run to finish ahead of both single-signal runs. Tightening any gate can change the sample. You need to see what changed, where it changed, and whether the difference survives another period.

This question sits next to the problem of counting one signal twice, but it isn't the same problem. Duplicate signals can inflate a score because they share inputs. Two genuinely different signals can still fail to complement each other. The formulas tell you where overlap might exist. The four-run test asks what the combination did in the historical sample.

Why can the combined gate fool you?

An AND rule creates an intersection. A company qualifies only if it clears both conditions, so the joint group can't be larger than either single-signal group. That mechanical narrowing can make a result look more selective without showing that the signals reinforce each other.

The joint version may drop most of the observations and leave one fortunate episode. It may remove an industry that struggled during the test window. It may favor larger companies because those companies have more complete filing histories. Or the cash-backing rule may serve as an accidental profitability filter when the intended claim was about earnings quality.

Read the selection change before naming the mechanism. Compare qualifying events, sector mix, market-cap distribution, and missing-data flags across the four runs. A better median outcome carried by several periods means something different from a better average pulled by one company. The portfolio line alone can't tell those stories apart.

Gate structure matters too. A strict gate says each condition is necessary. A score gate says one strength can compensate for another weakness. Strict gates and score gates express different claims, so don't switch between them during this test. Keep the combination rule fixed and change only whether each signal participates.

Build the four-run comparison before looking at returns

Name the two claims in plain language. Then write down the versions you'll run:

1. Baseline: neither signal active.
2. Value only: value active, cash backing bypassed.
3. Cash backing only: cash backing active, value bypassed.
4. Joint: both signals active.

The baseline isn't meant to become a usable thesis. It shows what the universe and backtest machinery produce before either stock-selection claim is imposed. The single-signal runs show each rule's standalone contribution. The joint run tests the intersection.

Freeze everything else. Use the same universe and test dates. Keep one benchmark, one rebalance cadence, one reporting lag, and one exit rule. Choose one event-study horizon before opening the reports. All returns in this comparison are hypothetical and exclude costs.

Pick the outcome in advance too. Median excess return at a fixed horizon is a sensible primary measure because it resists one extreme observation. Excess return is the selected company's return minus its benchmark's return over matching dates. Keep mean excess return, hit rate, drawdown, and event count beside it as diagnostics. Don't swap the primary measure when another column tells a nicer story.

Keep the thesis changes mechanical

Use parameters as switches so the formulas and null handling stay identical across versions. The shape below is illustrative, and the bundled templates are the reference for exact fields. The Quantery thesis documentation covers the full DSL.

# Value x cash-backing test: illustrative
params:
  use_value: 1
  use_cash_backing: 1
  yield_floor: 0.05
  conversion_floor: 0.80

features:
  fcf_ttm: ttm(free_cash_flow)
  ni_ttm: ttm(net_income)
  fcf_yield: if(market_cap > 0, fcf_ttm / market_cap, null)
  cash_conversion: if(ni_ttm > 0, fcf_ttm / ni_ttm, null)

criteria:
  value:
    rules:
      - { when: "$use_value == 0", score: 1 }
      - { when: "is_null(fcf_yield)", score: 0, flag: no_yield }
      - { when: "fcf_yield >= $yield_floor", score: 1 }
      - { else: 0 }
  cash_backing:
    rules:
      - { when: "$use_cash_backing == 0", score: 1 }
      - { when: "is_null(cash_conversion)", score: 0, flag: no_conversion }
      - { when: "cash_conversion >= $conversion_floor", score: 1 }
      - { else: 0 }

gate:
  mode: strict

Set both switches to zero for the baseline. Turn on one at a time for the standalone runs, then set both to one for the joint run. Save each state as its own thesis version and run the same backtest settings against each one. Quantery records the version behind a run, so the comparison doesn't depend on remembering which box was checked last Tuesday.

The bypass rules award a passing score when a signal is switched off. That's deliberate. Removing a criterion from the YAML entirely would also work, but it creates more room for an unrelated edit to slip between versions. One stable specification with two named switches makes the research log easier to audit.

Read the interaction as a difference of differences

Label the chosen outcome from the baseline R00, value-only R10, cash-only R01, and joint R11. Then calculate this diagnostic:

interaction = (R11 - R10) - (R01 - R00)

The first bracket asks what cash backing added after the value rule was active. The second asks what cash backing added without the value rule. If the difference is near zero, the cash rule's contribution didn't depend much on value in this sample. A positive difference supports complementarity. A negative difference says the pair contributed less than their standalone changes suggested.

Don't turn that arithmetic into false precision. Portfolio returns compound, the four selections overlap, and gate changes can alter entry dates. The contrast is a diagnostic for where to investigate. It isn't a causal estimate with a clean standard error attached.

Read the event study beside it. If the interaction appears in the mean but disappears in the median, inspect the extreme events. If it appears only at a long horizon with few completed observations, wait for more evidence. If the direction holds at nearby horizons and across separate periods, the combination has a better claim to being structural.

Inspect the companies that created the difference

The most useful comparison is often between the value-only group and the joint group. Which companies did cash backing remove? Open those dropped cases and inspect the exact criterion values and filing inputs.

You might find that the joint rule removed businesses with rising receivables and weak conversion. That fits the proposed mechanism. You might find that it mostly removed capital-heavy companies during an investment cycle. Then the ratio may be penalizing reinvestment, and the thesis needs a narrower claim. You might find that null values caused the removals. That's a coverage result wearing an economic costume.

Now compare the cash-only group with the joint group. Does value remove expensive companies across several sectors, or does it reduce the list to one industry? Check the overlap by period, not just across the whole backtest. A pair can look diversified in aggregate while each rebalance leans on a narrow cluster.

This inspection also tells you whether one rule belongs in the gate. If cash backing catches a failure mode you consider disqualifying, keeping it as a strict condition may make sense even when the interaction contrast is dull. If it grades evidence instead, use a score and test the score separately. The backtest grades the rule you wrote. It doesn't decide what claim you meant.

Protect the last comparison from your own search

Two switches are manageable. Threshold tuning multiplies the search fast. Once you try several yield floors, several conversion floors, alternate horizons, and different universes, the best interaction can be the lucky tail of your attempts. Bailey and coauthors explain why trying more strategy configurations raises backtest-overfitting risk.

Predeclare the first four runs. Keep the thresholds round. Record every version, including the flat and reversed results. If the joint signal looks useful, repeat the exact comparison in a holdout period that didn't help choose the rules. Then nudge one threshold at a time, following the same perturbation discipline used to test backtest robustness.

A combination earns its complexity when the joint contribution survives those checks and the dropped companies fit the mechanism you wrote down. If the apparent edge vanishes outside one interval, keep the simpler rule. If both signals work but don't interact, keep them as separate claims and weight them accordingly. If the pair exposes a distinct failure mode, preserve the pair and its audit trail.

That's the advantage of editable thesis rules. You don't have to trust that value and cash backing belong together because the pairing sounds sensible. Switch each one off, run the missing comparisons, and make the combination show its work.

Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.

← All articles