← Quantery Blog

How to test metric definition risk | Quantery Blog

October 22, 2026 · 8 min readthesis-craftaccountingbacktesting

A stock thesis has metric definition risk when a reasonable change to an accounting formula changes which companies pass. Test it by writing down the plausible versions of each important metric, changing one definition at a time, and comparing both the selected companies and the backtest. If the conclusion survives, the thesis is measuring an economic idea. If it breaks, the formula was doing more work than the idea.

This is different from moving a threshold. Raising a return-on-capital hurdle asks whether the cutoff is brittle. Replacing one defensible return-on-capital formula with another asks whether you measured the business consistently in the first place. Both tests matter, but they diagnose different faults.

Why can one metric have several valid definitions?

Accounting statements report line items. Most ratios used in screens are constructions laid over those line items. The construction depends on the question.

Take return on capital. The numerator might be operating income because you want profit before financing. It might use EBITDA when depreciation makes businesses hard to compare. The denominator might be total assets less current liabilities, or working capital plus fixed assets. An analyst studying acquisition-heavy companies may remove goodwill; another may keep it because management really spent that money.

None of those choices is neutral. Removing goodwill asks how productive the current operating assets are. Keeping it asks how productive all the capital management deployed has been, including acquisition premiums. The formulas can disagree while both remain arithmetically correct.

The same problem appears all over a value screen:

• Free cash flow can mean operating cash flow less all capital spending, or an estimate that subtracts only maintenance spending.
• Net debt can subtract every dollar of cash, or leave aside cash needed to run the business.
• Earnings yield can put operating income over enterprise value, EBITDA over enterprise value, or net income over market value.
• Book value can include goodwill and other intangibles, or remove them to produce tangible book value.

A vendor label hides these choices. A thesis built from raw fields makes them visible, which means you can test them instead of inheriting them.

Which definitions deserve a rerun?

Don't generate every formula you can imagine. Use alternatives that a careful reader could defend before seeing the result.

Start with the economic claim. If the claim is that a business earns strong returns on the capital required to operate, compare formulas that differ on what "required to operate" means. A version that excludes goodwill is plausible. A version that subtracts whichever balance-sheet line happens to improve the backtest isn't.

Then inspect where the chosen formula relies on judgment. Four questions expose most of the risk:

• Does the numerator cross accounting levels, such as pairing equity earnings with a firm-value denominator?
• Does the denominator contain balances that can be negative or close to zero?
• Does the formula remove a real cost because it is non-cash in the current period?
• Does it treat all industries as though their balance sheets work the same way?

That last question is why banks often need their own definitions. Debt is an operating input for a bank in a way it isn't for a manufacturer. A generic net-debt ratio can turn a business-model difference into an apparent warning.

Write the alternatives down before running them. This is the same discipline as setting stock-screen thresholds from an economic anchor. A definition chosen after looking at the winning curve is just another fitted parameter, even if it arrives dressed as accounting judgment.

Change one formula and keep the rest fixed

A clean definition test is an ablation: replace one component while everything else stays put. Don't change the formula, threshold, universe, and holding period in the same run. You won't know which change moved the result.

Suppose a quality-and-value thesis ranks companies on return on capital and earnings yield. Begin with a base version. Copy it, change only the capital definition, and give the variant a distinct meta.name that says what changed. Run both against the same universe on the same UTC date without refreshing data between them. Quantery stores one scan per thesis name and UTC date, so another version under the base name would replace the earlier same-day result.

Compare more than the headline return. Start with membership:

• Which companies survive both versions?
• Which appear only after goodwill is removed?
• Do the disagreements cluster in acquisitive industries?
• Are a few companies with tiny denominators jumping to the top?

That comparison tells you what the formula means in practice. If removing goodwill admits mostly serial acquirers, the variant is making a favorable choice about acquisition premiums. You may still prefer it, but now the preference has a visible consequence.

Because those separate names are separate theses, Quantery's Compare panel won't put these same-day results side by side. Open each stored run and save its report, then compare the memberships outside the panel. The two-run comparison workflow is for runs of one thesis on distinct UTC dates. It can reveal added and dropped names after a version change, but fresh prices and filings can also move the result. Either way, read several disagreements against their filings. A formula test without row-level inspection can tell you that results moved; it can't tell you why.

A definition test in the thesis DSL

The shape below is illustrative, and the bundled templates are the reference for exact fields. The thresholds are proposed research parameters, not claims about universal cutoffs.

# "Return on capital under two defensible definitions" - illustrative
meta:
  name: return-on-capital-definitions
  label: Return on capital definitions
  description: Compare full and tangible capital in a quality-and-value thesis.
  scale: { min: 0, max: 2 }

universe:
  exchanges: [NYSE, NASDAQ, AMEX]
  min_market_cap: 0
  exclude_sectors: []

quality:
  stale_months: 9
  required:
    ebit_ttm: no_operating_income
    market_cap: no_market_cap

params:
  roc_strong: 0.20
  roc_ok:     0.10
  yield_ok:   0.05

features:
  ebit_ttm:          ttm(operating_income)
  assets_now:        newest(total_assets)
  current_liab_now:  newest(current_liabilities)
  goodwill_now:      coalesce(newest(goodwill), 0)
  intangibles_now:   coalesce(newest(intangibles), 0)
  capital_full:      assets_now - current_liab_now
  capital_tangible:  capital_full - goodwill_now - intangibles_now
  roc_full:          if(capital_full > 0, ebit_ttm / capital_full, null)
  roc_tangible:      if(capital_tangible > 0, ebit_ttm / capital_tangible, null)
  debt_now:          newest(total_debt)
  cash_now:          newest(cash)
  enterprise_value:  market_cap + debt_now - cash_now
  earnings_yield:    if(enterprise_value > 0, ebit_ttm / enterprise_value, null)

criteria:
  capital_return:
    rules:
      - { when: "is_null(roc_full)", score: 0, flag: no_roc }
      - { when: "roc_full >= $roc_strong", score: 2 }
      - { when: "roc_full >= $roc_ok", score: 1 }
      - { else: 0 }
  valuation:
    rules:
      - { when: "earnings_yield >= $yield_ok", score: 2 }
      - { when: "earnings_yield > 0", score: 1 }
      - { else: 0 }

gate:
  mode: strict

The base thesis scores roc_full. The separately named variant changes only the three references needed to score roc_tangible. Keep the value rule, universe, quality policy, and backtest settings fixed. Each backtest receives its own record, so you can run both over the same historical dates and compare their reports without the same-day scan limitation.

The denominator guards matter. When capital is zero or negative, return on capital doesn't become wonderfully high or terribly low. It becomes a ratio without a useful economic reading, so the feature returns null. Put broken ratio handling inside the formula test before any winners appear.

You could put both definitions into one composite score. Resist that at first. Averaging them can hide disagreement. Separate runs force the disagreement into view.

What should the backtest comparison measure?

A definition-robust thesis should tell roughly the same story under each plausible formula. That doesn't require identical names or returns. Accounting choices have consequences. You are looking for continuity in the result's shape.

Check whether qualification events remain spread through time, whether performance depends on the same market regime, and whether one variant introduces a sector or size concentration. Compare the event study as well as the simulated portfolio. Every backtest is hypothetical and excludes costs, so a narrow advantage in one version isn't a verdict.

Membership stability is often more informative than a small return difference. If most qualifying companies overlap and the variants rank the shared names similarly, the thesis probably isn't hostage to the formula. If the lists barely touch, you have two theses with one label.

Next, inspect disagreement cases. A goodwill-heavy acquirer may rank well only under the tangible-capital version. A capital-intensive company may look strong under EBITDA and ordinary under operating income because depreciation is a real cost of maintaining its asset base. Those aren't data errors. They're clues about what each definition rewards.

Run the test across more than one period before settling on a version. A formula can look harmless in one regime because the balance-sheet feature it treats differently wasn't important then. A backtest result needs robustness checks, and a definition change is one of the strongest because it attacks the measurement itself.

What if the thesis fails the definition test?

Don't search for the version with the best return. Change the role of the metric.

A formula-sensitive measure can move from a hard gate to a score, where another criterion can offset it. It can become an annotation that tells you which filing deserves a closer read. You can split the universe so the definition is applied only where its economics make sense. Or you can remove it and ask whether the simpler thesis loses anything useful.

Sometimes the right outcome is two named theses. Return on tangible capital and return on total deployed capital answer different questions. Keeping them separate is clearer than pretending one is the canonical formula.

Quantery's documentation provides a thesis overview, while the bundled templates are the exact reference for fields and options. Save each definition under a distinct thesis name, run both backtests over identical dates, and keep both reports in the research record. For a same-day membership check, save each scan report and compare the lists outside Quantery. The point isn't to discover the official formula. There isn't one. The point is to learn whether your conclusion belongs to the business or to the arithmetic you chose.

Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.

← All articles