Property-Based Testing for Robust Softmax and Cross-Entropy

Muhammad MaazarXiv 2026 · arXiv

TL;DR

Standard machine learning libraries like PyTorch and JAX contain subtle numerical bugs in their softmax and cross-entropy implementations that can lead to vanishing or incorrect gradients. This paper demonstrates how to rediscover these bugs using property-based testing and proposes a more robust fix that handles extreme edge cases—like causal masking in Transformers—where previous solutions fail.

What Problem Does This Paper Address?

Softmax and cross-entropy are the fundamental building blocks of modern deep learning. To prevent numerical overflow, libraries use the "max-shift trick," subtracting the maximum value from logits before exponentiating them. However, recent research by Barron et al. (2026) revealed that even with this trick, two issues persist:

  1. Softmax Collapse: As a model becomes highly confident, the gradient of the hard-target cross-entropy incorrectly rounds to zero.
  2. Softmax Cancellation: The gradient of soft-target cross-entropy fails to reach zero even when student and teacher logits are identical.

While Barron proposed a fix, this paper identifies a remaining "edge of the edge case" where that fix produces NaN (Not a Number) gradients, particularly in scenarios involving extreme logit differences or causal attention masking.

Methods

The author employs Property-Based Testing (PBT) using the Python library Hypothesis. Unlike traditional unit tests that check specific inputs, PBT defines mathematical properties that should always be true and lets the framework search for inputs that break them.

The paper tests two core properties:

  • Property 1: Every row of a softmax Jacobian must sum to zero (in exact arithmetic).
  • Property 2: The gradient of the cross-entropy should be exactly zero when the student and teacher logits are identical.

To fix the identified failures, the author proposes a mathematical refinement. While Barron’s fix used a combination of softplus and logsumexp, the author simplifies this to a log1p (log(1+x)) approach:

  • Barron's approach: normalizer = softplus(logsumexp(others))
  • Author's approach: normalizer = log1p(exp(others).sum())

Datasets

This paper does not use traditional training datasets. Instead, it uses the Hypothesis library to procedurally generate tens of thousands of random logit vectors (lists of floats) to stress-test the numerical stability of the functions.

Key Insights and Ideas

  • PBT as a Debugging Powerhouse: You don't need a state-of-the-art LLM to find these bugs. Simple, automated property tests can rediscover complex numerical issues that have existed in major libraries for years.
  • The Precision Trap: The paper calculates exactly when softmax collapse occurs based on floating-point bits. For float32, the collapse happens when the margin between the top logit and others exceeds ~16.636.
  • The "All $-\infty$" Failure: Barron’s fix fails when max-shifting results in a vector of all $-\infty$ (except the max logit which is set to 0). Taking the logsumexp of an all $-\infty$ vector results in $-\infty$, but its automatic differentiation produces NaN.
  • Real-World Relevance: This isn't just theoretical. Causal masking in Transformers explicitly uses $-\infty$ to prevent looking at future tokens. A robust softmax must handle these "masked out" values without breaking the gradient flow.

Key Results

The author's implementation successfully passes the property tests where both standard PyTorch and Barron's proposed fix fail.

Specifically, for an extreme input vector like $[10^{308}, -10^{308}]$, the standard implementation gives zero gradients (collapse), and Barron's fix gives NaN gradients. The author's log1p fix provides the mathematically correct result.

Table 1: Forward softmax probabilities and computed Jacobians for [10308, −10308], evaluated in float64 on CPU using PyTorch 2.7.1.

Limitations and Future Work

The paper focuses specifically on the softmax and cross-entropy pair. While these are critical, other high-level functions in ML libraries might harbor similar numerical instabilities that haven't been rigorously tested via PBT.

The author suggests that property-based testing should be more widely adopted in the core test suites of libraries like PyTorch, JAX, and NumPy to catch these "silent" numerical errors before they impact model training in production.

Summary updated

Abstract

A recent paper of Barron et. al. (2026) identified two numerical issues with the softmax and cross-entropy functions in virtually all standard machine learning libraries (e.g., PyTorch and JAX). We show that these issues can be recovered through simple property-based tests in Hypothesis. However, our tests identify an additional edge case where their implementation fails. We then propose an even more robust softmax implementation which passes these tests and also supports use in causal attention masking. We compare our implementation with theirs and discuss some of the differences. Overall, this work shows that property-based testing can be a powerful tool for discovering and fixing numerical issues in machine learning libraries. Our implementations and property-based tests can be found at https://github.com/mmaaz-git/softmax-fix.
BibTeX
@misc{maaz2026property,
  author = {Muhammad Maaz},
  title = {{Property-Based Testing for Robust Softmax and Cross-Entropy}},
  year = {2026},
  eprint = {2610.36427},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36427},
}
Figure