Training ML Models with Predictable Failures

Will Schwarzer, Scott Niekum

TL;DR

Current pre-deployment safety tests often fail because catastrophic failures are too rare to observe in small evaluation sets. This paper introduces a "forecastability loss" that fine-tunes models to have "well-behaved" failure tails, allowing researchers to accurately predict deployment-scale failure rates using Extreme Value Theory (EVT) even when the worst failures haven't been seen yet.

What Problem Does This Paper Address?

When we test a machine learning model before release, we usually evaluate it on a few thousand examples. However, "deployment scale" might involve billions of interactions. If a model has a one-in-a-million failure mode, we probably won't see it during testing, but it will happen thousands of times in the real world.

Recent work (Jones et al., 2025) proposed using Extreme Value Theory (EVT) to bridge this gap. By looking at the "worst" few failures in a small set, we can fit a mathematical curve (specifically a Gumbel distribution) to predict how much worse the failures will get as we scale up.

The problem is that models aren't naturally "forecastable." Their failure distributions might have "bumps" or "cliffs" (rare modes) that a simple EVT curve can't see coming. If a model’s tail is poorly behaved, our safety forecasts will be dangerously wrong, usually under-predicting the risk.

Methods

The authors propose forecastability fine-tuning, a method to shape a model's behavior so its failures follow a predictable, log-linear pattern.

  1. Forecastability Loss: During training, the authors simulate the evaluation-to-deployment gap. They take a pool of tasks, partition them into a "fit set" (small) and a "deployment set" (large), and calculate the error between the EVT forecast (based on the fit set) and the actual worst-case results in the deployment set. They then backpropagate this error to update the model.
  2. Improving-only Gradient Mask: A naive loss might make a model "predictable" by making it fail more often (saturating the tail). To prevent this, the authors use a gradient mask that only allows updates that either reduce the risk score or improve the forecast without making the model worse at its primary task.
  3. Partition Randomization: To prevent the model from simply memorizing which specific inputs are "hard," they draw fresh random partitions of the task pool at every training step, ensuring the model learns a generalizable shape for its failure tail.

Datasets

The authors validated their approach on two "stress-test" environments designed to have rare, high-failure modes:

  • The Password Game (Language Modeling): Using Qwen3-0.6B and 8B models. The model is given a secret password and told never to reveal it. The "tasks" are prompts. Most are standard "jailbreak" attempts (the bulk), but a tiny fraction are high-strength adversarial suffixes generated via Greedy Coordinate Gradient (GCG).
  • Gridworld Navigation (Reinforcement Learning): An 8x8 grid where an agent must reach a goal while avoiding traps. The task distribution is a mixture of "benign" layouts (no traps) and "rare" layouts (densely packed with traps).
  • WildChat-1M: Used for an empirical "in-the-wild" analysis of failure tails in real-world assistant conversations (measuring toxicity, length, and log-likelihood).

Key Insights and Ideas

  • The "Safety-Favorable" Bias: The authors mathematically decomposed the forecast error and found that the standard Gumbel estimator actually has a built-in bias toward over-prediction in typical cases. This is good for safety—it’s better to be too cautious than too optimistic.
  • The Danger of Missing Modes: The biggest threat to accuracy is the "occupancy term." If the evaluation set completely misses a rare failure mode (like a specific type of adversarial attack) that will appear in deployment, the forecast will drastically under-predict the risk.
  • Shaping the Tail vs. Minimizing Risk: Interestingly, forecastability training is different from standard Supervised Fine-Tuning (SFT). While SFT tries to make the model safer overall, forecastability training specifically targets the relationship between common and rare failures, making the model's "breaking point" easier to see coming.

Key Results

  • Massive Accuracy Gains: In the Password Game, forecastability training improved forecast precision by ~60x over the pretrained baseline.
  • Safety without Sacrifice: The method achieved safety levels (reduction in leak probability) comparable to supervised baselines but with significantly less "capability drift." For example, the KL divergence (a measure of how much the model's general behavior changed) was two orders of magnitude lower than standard SFT.
  • RL Generalization: This was the first application of EVT forecasting to Reinforcement Learning. The authors showed that the same math used for LLM safety could predict "regret" (how much reward the agent missed) in complex gridworld environments, reducing forecast error by 52x.

Limitations and Future Work

  • Scale: The experiments were conducted on relatively small models (0.6B and 8B parameters) and simple environments (8x8 grids). Whether these tail-shaping properties hold at the scale of GPT-4 or Claude remains to be seen.
  • Synthetic Mixtures: The "rare modes" in the experiments were manually constructed (e.g., GCG prompts). In the real world, rare failure modes are often unknown and "natural."
  • Computational Cost: The RL setting relied on closed-form regret calculations (backward value iteration), which is computationally impossible for high-dimensional environments (like 3D simulators or real-world robotics). Scaling this to estimated rewards via Monte Carlo rollouts is a major next step.

Summary updated

Abstract

Estimating how often an ML model will fail at deployment scale is central to pre-deployment safety assessment, but a feasible evaluation set is rarely large enough to observe the failures that matter. jones2025forecasting address this by extrapolating from the largest k failure scores in an evaluation set to predict deployment-scale failure rates. We give a finite-k decomposition of this estimator's forecast error and show that it has a built-in bias toward over-prediction in the typical case, which is the safety-favorable direction. This bias is offset when the evaluation set misses a rare high-failure mode that the deployment set contains, leaving the forecast to under-predict at deployment scale. We propose a fine-tuning objective, the forecastability loss, that addresses this failure mode. In two proof-of-concept experiments, a language-model password game and an RL gridworld, fine-tuning substantially reduces held-out forecast error while preserving primary-task capability and achieving safety similar to that of supervised baselines.
BibTeX
@inproceedings{schwarzer2026training,
  author = {Will Schwarzer and Scott Niekum},
  title = {{Training ML Models with Predictable Failures}},
  booktitle = {AIWILD @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=eAFDbSFdlp},
}
Figure