Beating seasonal naive is not the bar
The gradient-boosting model in my NHS A&E project beat the seasonal-naive baseline by 28 to 35% on point error, across all three targets. On the same 73 forecast origins it produced 90% intervals that covered 83% of outcomes one month ahead and 76% six months ahead, and it scored worse than exponential smoothing on the interval metric I had named as the selection criterion before seeing any results. A write-up that stopped after the first sentence would be accurate and would mislead a bed manager. This post is about why the second sentence is the one that decides what to ship.
Two questions, two metrics
“Did it beat the baseline?” is usually answered with a point-error metric. Here it was MASE, the mean absolute error scaled by the seasonal-naive error over the training period, so that every provider is compared on the same footing. The gradient-boosting model’s MASE came in 28 to 35% below seasonal naive’s, depending on the target. That is a real improvement, and it clears the bar most published comparisons set.
“Can I plan on the range?” is a different question and needs a different metric. If a forecast says the peak month will bring between 9,000 and 11,000 admissions with 90% confidence, the plan depends on both numbers and on the 90% being true. The weighted interval score (WIS) rewards a forecast for being sharp and penalises it for outcomes that fall outside the interval, in proportion to how far outside they fall. It is a proper scoring rule: a forecaster minimises it by reporting what they believe, and cannot game it by hedging or by over-claiming.
On WIS the gradient-boosting model was level with ETS on all-type attendances, 7% worse on Type 1 attendances and 21% worse on admissions. Its intervals were too narrow. The shortfall was worst in 2020, where coverage fell to 65%, and smallest in 2025, where it reached 92%. The model had learned the level of the series well and its uncertainty badly.
Why the naive bar is too low
Seasonal naive is the right first baseline because it is the forecast a competent person makes with no model at all. For most monthly operational series it is also a weak one. Any method that tracks the level, whether ETS, ARIMA or a regression on the last twelve months, will beat it comfortably, so “beats seasonal naive by 30%” tells a buyer the model is not broken. It does not tell them whether the model is better than the classical method their own analyst could fit in an afternoon.
That classical method is the second baseline, and commercially it is the one that matters. If a model with a feature pipeline, a tuning loop and a conformal calibration step cannot beat ETS on the metric the decision uses, the buyer should have ETS, and the model should stay in the repository as the thing to improve. That is what the NHS project concluded, and what its page says.
What did not help
Two things a reader might expect to rescue an under-performing model were tried.
Hyperparameter tuning: sixteen configurations, searched on forecast origins before the backtest window so that the search could not see the test data. The spread across the whole search was 9% and the winner sat close to the defaults. Tuning is not why the model trails ETS.
A longer calibration window for the conformal intervals: 24 months instead of the default gave better coverage, as expected, and worse WIS, because the wider intervals cost more in sharpness than they recovered in coverage. The pre-registration had named WIS as the criterion, so I did not adopt the wider window. Had I chosen coverage as the criterion instead, the decision would have gone the other way, which is why the criterion has to be fixed before the results are in and not after.
How to read a forecast claim
Four questions, in order. Which baseline, and was the classical one included? Which metric, and does it score the range or only the centre? Was the metric chosen before or after the results? And on what data: today’s revised file, or the file that existed at each forecast origin?
On the last question the NHS project found, against my expectation, that revisions made no material difference: as-of and final-data scores were within 2% for every model and the ranking did not change. I would have skipped the vintage archive if I had trusted the expectation. The measurement says the archive was unnecessary here, and I only know that because I built it.
Code, pre-registration and results for every number above are in the project repository.