Abstract
Deep neural networks are notoriously overconfident, especially when encountering data that deviates from their training distribution (Out-of-Distribution shift). This paper introduces MaxEnt Loss, a novel regularization technique based on the Principle of Maximum Entropy. By utilizing statistical constraints to prevent overconfidence, MaxEnt Loss delivers state-of-the-art calibration performance across various benchmarks without sacrificing predictive accuracy.
Introduction: The Calibration Crisis
In machine learning, calibration measures how well a model’s predicted probabilities reflect reality. Ideally, if a model assigns a 90% probability to a classification, it should be correct 90% of the time. However, modern deep networks are frequently "uncalibrated," tending toward extreme overconfidence even when making incorrect predictions.
This overconfidence is particularly dangerous in safety-critical domains like autonomous driving or medical diagnostics. While existing techniques can calibrate models for In-Distribution (ID) data, they often fail under Out-of-Distribution (OOD) shifts—such as environmental changes or rare edge cases. MaxEnt Loss addresses this by keeping models "honest" about their uncertainty as the environment shifts.
Methodology: Maximum Entropy with Constraints
The proposed method leverages the Principle of Maximum Entropy (MaxEnt). In information theory, the distribution with the highest entropy is the one making the fewest assumptions (the uniform distribution). By maximizing entropy, the model is discouraged from producing "peaky," overconfident probability distributions.
To maintain classification utility, the authors introduce statistical constraints derived from the training data:
- Mean Constraint: Constrains the expected average of the predicted distribution.
- Variance Constraint: Constrains the spread or "width" of the distribution.
- Mean + Variance: A combined approach utilizing both statistical moments.
A key innovation is the use of Lagrange multipliers, solved numerically via the Newton-Raphson method before training. This automates the balancing of the entropy term with standard classification loss, removing the need for manual hyperparameter tuning.
Experimental Evaluation
The method was evaluated on six major benchmarks testing robustness against distribution shifts:
- Synthetic OOD: CIFAR-10-C, CIFAR-100-C, and TinyImageNet-C, featuring 19 different corruptions (e.g., noise, weather) at varying intensities.
- Real-world OOD (Wilds Benchmark): Including Camelyon17 (medical imaging), iWildCam (wildlife monitoring), and FMoW (satellite imagery).
Key Results and Insights
MaxEnt Loss achieved state-of-the-art calibration across nearly all OOD benchmarks, measured by Expected Calibration Error (ECE).
- Superior Calibration: On CIFAR-100-C, Mean-constrained MaxEnt reduced ECE to 6.30%, significantly outperforming standard Cross-Entropy (10.2%) and Focal Loss (8.8%).
- No Accuracy Trade-off: Unlike many calibration methods, MaxEnt Loss maintains or slightly improves top-1 accuracy while drastically reducing overconfidence.
- Feature Norm Correlation: The authors observed that better-calibrated models exhibit smaller L2 feature norms in the penultimate layer. MaxEnt Loss naturally encourages these smaller norms, providing a buffer against the "feature explosion" typically associated with OOD overconfidence.
Conclusion and Future Work
MaxEnt Loss provides a robust, "plug-and-play" solution for model calibration under pressure. It serves as a superior starting point that can be further enhanced by post-hoc methods like Temperature Scaling. Future research directions include exploring unique Lagrange multipliers for different constraints and expanding the framework to non-image domains such as Natural Language Processing.
Summary updated