quant-validation

Files (1)

• SKILL.md

SKILL.md (first lines)

# Validating a financial model — the five ways the number lies

A backtest that looks excellent and loses money live is not usually a bad
strategy. It is a good measurement of the wrong thing. Each section below is one
mechanism by which a number becomes convincing without becoming true.

**On sourcing.** The methods here are standard and attributable — most of them to
Marcos López de Prado's *Advances in Financial Machine Learning*, with the
information-ratio framing from Grinold & Kahn. This file states the MECHANISM and
what to check, and deliberately does not restate formulas from memory. Where an
implementation needs an exact expression — the deflated Sharpe ratio in
particular — verify it against the primary source before shipping a number that
depends on it. A formula recalled approximately is worse here than no formula:
it produces a specific, wrong, confident figure.

## 1. Purged cross-validation with an embargo

**The leak.** In a normal k-fold split, training and test rows are disjoint. In a
financial series they are not independent: a label at time *t* is computed from
data spanning *t* to *t+h*. A training observation inside that window has seen the
future the test observation is being asked to predict.

**Purging.** Drop from the training set every observation whose label window
overlaps the label window of any test observation. Not the observation's
timestamp — its *label window*. This is the step people skip, because a plain
timestamp split looks like it already separates them.

**The embargo.** Purging is not enough when features are serially correlated: a
training row immediately AFTER the test set still carries information about it.
Drop a further band after each test fold. The band is a fraction of the total
sample; there is no universal value, so state the one used and why.

**Combinatorial purged CV.** A single train/test split yields one backtest path
and one Sharpe. Splitting combinatorially yields many paths and therefore a
*distribution*, which is what you actually want: a strategy whose single path
looks good and whose distribution straddles zero has told you something a point
estimate hid.

**What to check:** is the split purged, is there an embargo, is its size stated,
and is the reported figure a distribution or a single draw.

## 2. Triple-barrier labelling

**The problem with fixed-horizon returns.** Labelling "the return over the next
five days" assumes you would have held for five days. You would not: a stop-loss
would have taken you out on day two. The model is trained on an outcome that
could not have happened.

**The method.** Three barriers per observation — a profit-take level, a stop-loss
level, and a time limit. The label is *which barrier was touched first*. Levels
are usually set from a volatility estimate rather than fixed, because a 2% move
means different things in different regimes.

**What to check:** are the barriers volatility-scaled, is the time limit stated,
and does the label record which barrier ended the observation rather than only
the sign.

## 3. Sample uniqueness under overlapping labels

**The problem.** Overlapping label windows mean two rows can describe largely the
same outcome. Standard learning assumes independent draws; here they are not, so
the effective sample is far smaller than the row count and every confidence
interval computed from that count is too narrow.

**Two responses:** weight each observation by its average uniqueness (how much of
its label window it does not share), or draw with a sequential bootstrap that
prefers observations overlapping little with those already drawn.

**What to check:** is a uniqueness weighting or effective sample size reported. A
row count offered as a sample size is a wrong number, not a rough one.

## 4. Fractional differentiation

**The dilemma.** Price levels are non-stationary; a model fitted to them learns a
level that will not recur. The reflex is a first difference — returns — which is
stationary and has thrown away the memory the signal lived in.

**The method.** Difference by the smallest order `d`, generally fractional, at
which the series passes a stationarity test while retaining maximum correlation
with the undifferenced series. `d` is a result, not a setting: it is searched for,
and it is reported.

**What to check:** is `d` reported at all, was it searched rather than assumed,
and was correlation with the original series measured — not just the stationarity
test passed. Passing the test is the constraint; keeping the memory is the
objective.

## 5. Meta-labelling

**What it is.** Two models rather than one. The primary decides the SIDE — long,
short, flat. The secondary decides only whether to ACT on that call, as a binary:
take this bet or pass.

**Why it helps.** The two tasks have different error costs. A side model tuned for
accuracy tends to trade too often; a secondary model can raise precision — fewer,
better-founded bets — without touching the side logic. It also gives a natural
place to size a bet by confidence, which a single model conflates with direction.

**What to check:** if a model both picks the side and decides whether to trade,
say whether those were separated. If not, the reported precision is measuring two
decisions at once.

## 6. The multiple-testing problem

**The mechanism.** Try enough configurations and one will look excellent by
chance. The reported Sharpe of the best of N trials is not an estimate of that
strategy's Sharpe — it is the maximum of N draws, and its expectation rises with
N even when every strategy is worthless.

**The minimum honest response:** report N. How many feature sets, parameter
values, and universes were tried to reach the reported one. A Sharpe without a
trials count cannot be interpreted, and the count is usually much larger than
people remember — every abandoned variant counts.

**The correction:** the deflated Sharpe ratio adjusts for the number of trials and
for the non-normality of returns. Its exact expression is not restated here (see
the sourcing note above); implement it from the primary source.

**What to check:** is N reported, and if a correction is claimed, does the

From avelikiy/great_cto, MIT.

More from avelikiy/great_cto