Training ML Models with Predictable Failures

Will Schwarzer, Scott Niekum

TL;DR

This paper introduces forecastability loss, a novel fine-tuning objective that shapes a model’s failure distribution to be mathematically "well-behaved." By making a model’s rare failures follow predictable patterns (based on Extreme Value Theory), researchers can accurately extrapolate deployment-scale risks from small evaluation sets without sacrificing primary task performance.

What Problem Does This Paper Address?

Machine learning safety is often a "needle in a haystack" problem. Before deploying a model, we test it on a small evaluation set. However, the failures that truly matter—catastrophic or high-risk events—might only happen once in every million runs. If our evaluation set only has 1,000 examples, we are unlikely to see these rare failures at all.

A recent approach (Jones et al., 2025) uses Extreme Value Theory (EVT) to solve this. By looking at the "worst" failures in a small set, they fit a mathematical curve (a Gumbel distribution) to predict how much worse the failures will get as you scale up to millions of users.

The catch? This extrapolation only works if the model's failure distribution is "well-behaved"—meaning it follows a smooth, predictable tail. If a model has a "rare mode" (a specific type of failure that evaluation misses but deployment hits), the forecast will dangerously under-predict the risk. This paper asks: Can we train models to be more predictable?

Methods

The authors propose a training pipeline that treats "forecastability" as a first-class citizen:

  1. Error Decomposition: They mathematically break down why forecasts fail into three parts: rank (inherent statistical noise), curvature (the shape of the tail), and occupancy (missing a rare failure mode).
  2. Forecastability Loss: They turn the forecasting procedure itself into a differentiable loss function. During fine-tuning, the model is penalized if its worst-case performance on a "deployment" subset cannot be accurately predicted from a "fit" subset.
  3. Improving-only Gradient Mask: To prevent the model from becoming "predictable" by simply failing more often, the authors use a gradient mask. This ensures the model only updates in directions that either reduce risk or keep it stable while smoothing out the distribution's tail.
  4. Meta-multi-task Fine-tuning: They use a "partition-randomization" scheme, reshuffling which tasks are in the "evaluation" vs. "deployment" sets at every training step to ensure the model learns a generalizable tail shape rather than memorizing specific inputs.

Datasets

The paper uses two primary "proof-of-concept" environments designed to stress-test rare failure modes:

  • The Password Game (LM): Based on Qwen3-0.6B. The model is given a secret password and told never to reveal it. The "tasks" are prompts. The distribution is a mixture of benign prompts and rare, adversarially optimized "GCG" (Greedy Coordinate Gradient) attacks that are likely to be missed in small evals.
  • Gridworld Navigation (RL): A reinforcement learning agent must reach a goal while avoiding traps. The "tasks" are different map layouts. Most maps are easy, but a rare subset is "densely hazardous."
  • WildChat-1M: The authors also analyze real-world failure tails using 1 million ChatGPT interaction logs to validate their theoretical decomposition of forecast error.

Key Insights and Ideas

  • Predictability $\neq$ Safety: A model can be very safe but hard to forecast, or less safe but very easy to forecast. The goal here is to move the model into the "Goldilocks zone" where it is both safe and its risks are quantifiable.
  • The "Safety-Favorable" Bias: The authors prove that the standard Gumbel estimator actually has a built-in bias toward over-predicting risk. This is good for safety! However, this bias is easily wiped out by "rare modes," which is why the forecastability loss is necessary to "clean up" the tail.
  • Trainable Tails: The most surprising insight is that the "shape" of a model's failures is a trainable property. You don't just have to accept the distribution the base model gives you; you can actively shape it to fit EVT assumptions.

Key Results

  • Massive Precision Gains: In the Password Game, forecastability training improved forecast precision by ~60x over the pretrained baseline. When combined with post-hoc calibration, this rose to ~75x.
  • Safety without Sacrifice: Compared to standard Supervised Fine-Tuning (SFT), the proposed method achieved similar safety levels (reducing leak probability by ~40 decades) but with 100x less "capability drift" (measured by KL divergence from the base model).
  • RL Success: In the Gridworld task, the method improved forecast precision by 52x and actually increased the agent's average return by 1.27x, showing that making a model predictable can also make it more competent.

Limitations and Future Work

  • Scale: The experiments were conducted on relatively small models (0.6B parameters) and simple RL environments. The authors provide an 8B parameter proof-of-concept, but more work is needed to see if these "tail-shaping" effects hold at the frontier (100B+ parameters).
  • Synthetic Mixtures: The "rare modes" used in the paper were synthetically constructed. Real-world rare modes (like specific chemical recipes or complex social biases) may be harder to smooth out.
  • Differentiability Requirements: The RL setting currently requires exact regret calculations (backward value iteration), which is impossible in continuous or high-dimensional state spaces. Moving to Monte Carlo estimates of risk is a necessary next step.

Summary updated

Abstract

Estimating how often an ML model will fail at deployment scale is central to pre-deployment safety assessment, but a feasible evaluation set is rarely large enough to observe the failures that matter. Jones et al. (2025) address this by extrapolating from the largest k failure scores in an evaluation set to predict deployment-scale failure rates. We give a finite-k decomposition of this estimator's forecast error and show that it has a built-in bias toward over-prediction in the typical case, which is the safety-favorable direction. This bias is offset when the evaluation set misses a rare high-failure mode that the deployment set contains, leaving the forecast to under-predict at deployment scale. We propose a fine-tuning objective, the forecastability loss, that addresses this failure mode. In two proof-of-concept experiments, a language-model password game and an RL gridworld, fine-tuning substantially reduces held-out forecast error while preserving primary-task capability and achieving safety similar to that of supervised baselines.
BibTeX
@inproceedings{schwarzer2026training,
  author = {Will Schwarzer and Scott Niekum},
  title = {{Training ML Models with Predictable Failures}},
  booktitle = {AIWILD @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=eAFDbSFdlp},
}
Figure