How to validate a football prediction model

Freeze the question before choosing the winner

Define the market, target horizon, feature cutoff, evaluation period and primary metrics before inspecting the final results. If these rules move after a disappointing test, the next score belongs to a new experiment.

Randomly splitting rows is often unsafe in sport because later observations can reveal team state, competition state or market information that did not exist earlier. Time-ordered evaluation and same-kickoff batching reduce that risk.

Use several independent lenses

Calibration asks whether stated probabilities match frequencies. Brier score and log loss measure probability error. Discrimination measures ranking. Market comparison asks whether the model adds information beyond prices. None is a substitute for the others.

Report the simple baseline and bookmaker benchmark under the same observation policy. A complex model that cannot beat a league-rate baseline out of sample has not earned a product layer.

Prospective evidence closes the hindsight gap

The final test records the decision and attainable price before kickoff, then settles every row. Coverage, withdrawals and failures stay visible. Short-run ROI remains descriptive until the probability and price evidence mature.

• Version every model artifact.
• Preserve raw-source hashes.
• Record the exact quote and timestamp.
• Publish cohort size and missing coverage.
• Separate changed methodologies.