How to validate a stock thesis before running it
To validate a stock thesis before running it, check two different things. First, make sure the specification is executable, meaning the engine can evaluate every line of it without guessing. Then make sure it expresses the business claim you intended. A parser can confirm that leverage is calculable. Deciding whether your definition belongs in the thesis takes judgment.
Do both checks before reading a survivor list (the companies that passed the thesis gate) or a backtest curve. Otherwise the output starts negotiating with you. An unexpected company invites a new exclusion. A pretty result makes a questionable proxy feel settled. Validation is cheaper when there isn't yet an outcome to defend.
What can mechanical validation prove?
Mechanical validation asks whether the thesis engine can evaluate the document without guessing. In Quantery, that means the YAML matches the thesis schema and every expression uses the supported language correctly. The product documentation gives an overview. For exact fields and structure, use the bundled templates and the validator's response.
A validator can catch an unknown field, a misspelled parameter, a feature used before it is declared, a malformed criterion rule, or pct_rank() placed where cross-sectional ranking isn't supported. It can also reject a period function whose first argument isn't a filing field. These aren't stylistic complaints. Any one of them makes the requested calculation undefined.
Validation does more than spare you a failed run. It fixes the meaning of the document. If newest(cash) compiles, the engine knows to read cash from the newest stored filing period. If $leverage_max resolves, the threshold exists in the parameter block. If a rule contains either a when condition and score or an else, the engine can follow a deterministic path.
That still leaves plenty unchecked. A valid expression can divide the wrong quantities. A valid rule can award its strongest score too early. A valid backtest can answer a question nobody meant to ask. Passing validation establishes that the document is runnable. The thesis itself still needs scrutiny.
Does each formula match the business sentence?
Write the claim beside the formula in plain language. For example: companies with operating profit that covers their current financing burden deserve a higher debt-service score. The corresponding feature might divide trailing operating income by trailing interest expense.
Now challenge the translation (the claim-to-proxy step that building a thesis worth testing walks through). What happens when interest expense is zero? What happens when operating income is negative? Does the ratio compare flows from the same window? If the claim is about current debt service, using a balance-sheet debt figure in the denominator would compile, but it would answer a different question.
Balance-sheet formulas need another discipline: keep values from the same statement together. newest(total_debt) and newest(cash) read both fields from the newest period. latest() can scan backward until it finds a non-null value, which is helpful for a standalone field but can splice debt from one quarter to cash from another. The result may describe a balance sheet the company never filed. The balance-sheet identity primer explains why period alignment matters.
Flows need compatible windows too. Revenue, operating income, and cash flow describe periods. Use trailing filed periods when the thesis makes a trailing claim. Trailing versus forward numbers covers why filed values and filing dates belong in a point-in-time test.
Keep related ideas in separate features while you're checking them. Operating margin says how much operating profit remains from revenue. Net debt over EBITDA says how large net borrowing is beside an earnings proxy. Combining them into one clever expression makes errors harder to spot and turns an awkward economic trade-off into arithmetic you didn't mean to endorse.
Will nulls and rule order change the answer?
Treat missing data as its own state. In the thesis language, arithmetic with null produces null, and comparisons involving null evaluate false. That behavior is safe only when you write rules with it in mind.
Suppose a profitability criterion begins with its strongest threshold and ends with a catch-all zero. A missing operating margin falls through to zero. The score may be acceptable, but the reason disappears. Put an explicit null guard first and attach a flag. Now the result says no_operating_margin instead of pretending the company failed an observed margin test.
Rule order matters because the first matching rule wins. Put the strongest condition first, then weaker conditions, then the fallback. Reverse the bands and a company that clears the strongest threshold gets captured by the weaker one before the engine reaches it. The document can be perfectly valid and still score every strong company incorrectly.
Defaults deserve suspicion. Some filing fields, such as dividends paid, may reasonably use zero when absence means no payout for the research question. Debt, revenue, or operating income shouldn't receive that treatment. A zero substituted for an unknown balance is evidence you created. Mark indispensable inputs under quality.required so an incomplete company can't collect points elsewhere and pass.
This is also where the gate design gets tested. A strict gate requires each criterion to clear its lower band. A score gate allows points to compensate. Strict gates and score gates encode different claims, so validation should include a sentence explaining why compensation is allowed or forbidden.
What does a validated example look like?
The shape below is illustrative, and the bundled templates remain the reference for exact fields. The thresholds are proposed research parameters. This version has been checked against the thesis compiler, which accepts its fields, expressions, rules, and backtest settings. Predictive value remains an empirical question.
meta:
name: operating-resilience
label: Operating Resilience
description: Profitable businesses with manageable net debt.
scale: { min: 0, max: 2 }
universe:
exchanges: [NYSE, NASDAQ, AMEX]
min_market_cap: 200000000
exclude_sectors: [Financial Services]
quality:
stale_months: 9
required:
price: no_price
revenue_ttm: no_revenue
params:
margin_strong: 0.15
margin_ok: 0.05
leverage_max: 2.0
features:
revenue_ttm: ttm(revenue)
ebit_ttm: ttm(operating_income)
ebitda_ttm: ttm(ebitda)
operating_margin: if(revenue_ttm > 0,
ebit_ttm / revenue_ttm, null)
debt: newest(total_debt)
cash: newest(cash)
net_debt: debt - cash
leverage: if(ebitda_ttm > 0,
net_debt / ebitda_ttm, null)
criteria:
profitability:
rules:
- { when: "is_null(operating_margin)", score: 0, flag: no_operating_margin }
- { when: "operating_margin >= $margin_strong", score: 2 }
- { when: "operating_margin >= $margin_ok", score: 1 }
- { else: 0 }
balance_sheet:
rules:
- { when: "net_debt <= 0", score: 2 }
- { when: "is_null(leverage)", score: 0, flag: no_leverage }
- { when: "leverage <= 1.0", score: 2 }
- { when: "leverage <= $leverage_max", score: 1 }
- { else: 0 }
gate:
mode: strict
backtest:
rebalance: monthly
reporting_lag_days: 1
entry: next_close
exit: { max_hold_months: 12, on_gate_fail: true }
benchmarks: [SPY]Read it as prose. The thesis requires a current price and a non-null trailing revenue feature. A non-null feature can still come from an incomplete trailing window, so check period coverage separately. The thesis scores operating margin, then scores net debt relative to positive EBITDA. A net-cash company receives the strongest balance-sheet score without needing a leverage ratio. A company with net debt and non-positive EBITDA gets no ratio and no passing point.
That last behavior comes from rule order. net_debt <= 0 is tested before the null leverage guard, because a net-cash balance sheet can be recognized even when EBITDA isn't positive. If you believe loss-making companies should fail regardless of cash, put the EBITDA test first. Both arrangements compile. Only one matches your sentence.
The backtest block also makes assumptions visible. It specifies monthly rebalancing, a short reporting delay after a filing arrives, entry at the next close, and an exit after gate failure or the maximum holding window. Those settings are part of the research claim. Don't validate the features and leave the clock implicit.
How do you test meaning without peeking at returns?
Before the full run, evaluate hand-built cases. You don't need real companies for this pass. Write down the expected result for a profitable net-cash business, a profitable indebted business below the leverage ceiling, a company with negative EBITDA, and a company missing cash. Walk each case through the rules from top to bottom.
This catches semantic mistakes that compilation can't. If missing cash makes net_debt null, the net-cash comparison is false and leverage stays null. The criterion reaches no_leverage and scores zero. Is that what you wanted? If yes, preserve it. If no, change the required fields instead of adding a rescue rule after a scan exposes a famous company.
Then inspect a small set of real evaluations and trace each surprise to the matched rule and filing inputs. The result-audit workflow shows how to follow a score down to its periods and sources. Use this pass to check whether the implementation behaves like the written claim. Leave performance for the backtest.
Keep the compiler check and the semantic checklist in the research log. When a later version changes leverage_max, the diff should show one changed assumption. If the same edit also reorders the rules, you won't know what caused a different result.
What still needs a backtest?
Validation establishes that the rules run and mean what you wrote. A walk-forward backtest using information available on each simulated date must evaluate whether companies selected by those rules had different subsequent outcomes.
Run the declared version first. Every result is hypothetical and excludes costs. Compare it with a simpler baseline, then change one parameter at a time. Try nearby round thresholds. Then split the period and ask whether one sector or a small group of names carried the outcome. What makes a backtest honest covers the timing and data constraints; testing whether a result is real covers perturbing the choices.
A failed backtest still gives validation a purpose: a correctly stated idea failed its historical test. A malformed thesis can't give you even that. Get the document compiling and the edge cases walked before any return figure exists. Then let the backtest judge the claim you actually wrote.
Research tooling, not investment advice. Nothing here is a recommendation to buy, sell, or hold any security. Screens, scores, and backtests are informational only; backtested results are hypothetical, exclude costs such as commissions and slippage, and do not guarantee future results. Verify against primary filings and make your own decisions.
Want to try this on your own rules? Quantery is free for 14 days: the full app, no card required.
← All articles