A result is not evidence unless it existed prospectively
Backtests are useful for rejecting bad ideas, but they are easy to overfit and impossible to experience at the historical price shown later. Our proof protocol begins before kickoff and preserves failures as carefully as successes.
Freeze the decision before the outcome
Every shadow row records its decision time, fixture kickoff, model artifact, feature cutoff, bookmaker, attainable price and raw-source hashes. A later rerun cannot overwrite it. Changing weights or logic requires a new methodology version and therefore a separate evaluation cohort.
This prevents the common but misleading practice of improving a method after seeing its misses while continuing to present the whole history as one unchanged system.
Use one fixed decision horizon
Evaluation selects at most one decision per fixture: the eligible snapshot closest to 90 minutes before kickoff inside a frozen lead-time window. It does not search the day for the price or prediction that later looked best.
Features with timestamps after the decision are rejected. Fixtures sharing the same kickoff are processed as one batch, so one match’s outcome cannot become another match’s input.
Separate prediction quality from betting variance
Brier score, log loss and calibration evaluate whether probabilities are honest. Same-bookmaker closing-line value evaluates whether the recorded price was competitive. Flat-stake ROI is retained as a descriptive outcome, but it is not allowed to carry the argument on a small sample.
Vendor probabilities, proprietary model versions and changed gate definitions live in isolated cohorts. Combining them would manufacture sample size that no single method actually earned.
Fail closed when price evidence is weak
A changing vendor row ID is not a market move. A single stale bookmaker is not consensus. A closing price must come from the same bookmaker, be captured before kickoff and carry a vendor timestamp no later than the capture.
The current feed fails production depth, freshness and historical-season requirements. The application therefore shows a Research Lock instead of converting incomplete evidence into a recommendation.
Statistical gates are necessary, not sufficient
- 500 settled target-horizon probability observations;
- 150 frozen-gate candidate selections;
- at least 80% same-bookmaker closing-price coverage;
- Brier skill above 2% with ECE no higher than 3%;
- a positive lower 95% bootstrap bound for mean CLV;
- independent methodology, security and jurisdiction review.