Reliable machine learning requires models to know when they are "out of their depth." While many out-of-distribution (OOD) detection methods exist, they often require access to the original training data or modifications to the training process. These are luxuries that practitioners using third-party, pre-trained models rarely have.
In their ICLR 2024 paper, Jelenić et al. introduce BLOOD (Between Layer Out-of-Distribution detection), a "white-box" method that identifies OOD data by measuring how smoothly a Transformer processes information between its internal layers.
TL;DR
BLOOD detects out-of-distribution data by measuring the smoothness of transformations between intermediate Transformer layers, requiring only model weights and no training data. It leverages the discovery that fine-tuning "smooths" the model's internal mappings for in-distribution data, while leaving out-of-distribution regions sharp and erratic.
The Challenge of White-Box OOD Detection
Most OOD detection research falls into two camps:
- Black-box methods that look only at final outputs (like softmax probabilities).
- Open-box methods that require access to the training set to calculate distances in the representation space.
However, the modern AI ecosystem relies heavily on white-box scenarios: practitioners have the weights of a massive pre-trained model (e.g., from Hugging Face), but lack the original training data. Existing white-box methods often struggle with "overconfidence," where a model gives a high-certainty prediction for an input it has never seen before. BLOOD aims to fill this gap by looking inside the model's transformation process rather than just at its final answer.
Methodology: Transformation Smoothness
The core idea behind BLOOD is transformation smoothness. If you take an input $x$ and its representation at layer $l$, how much does that representation change when passed to layer $l+1$?
Mathematically, the authors quantify this using the square of the Frobenius norm of the Jacobian matrix. A high norm indicates a "sharp" transformation (small changes in input lead to large changes in output), while a low norm indicates a "smooth" one.
Because calculating a full Jacobian for modern high-dimensional Transformers is computationally prohibitive, the authors developed an unbiased estimator. By using Jacobian-vector products—which are efficient in modern auto-differentiation libraries—they can estimate the smoothness score using a small number of random projections ($M=50$ in their tests).
The authors propose two versions of their metric:
- $BLOOD_M$: The average smoothness score across all layers.
- $BLOOD_L$: The smoothness score of the very last layer transformation.
Key Insights and Ideas
- Training as a "Smoothing" Process: The most profound insight is that the learning algorithm focuses on the in-distribution (ID) region of the representation space. As the model learns, the transformations between layers become significantly smoother for ID data.
- OOD Sharpness: Because the model never sees OOD data during training, the transformations for those inputs remain "sharp"—similar to the erratic mappings of a randomly initialized model.
- Complexity Improves Detection: Counter-intuitively, BLOOD performs better on complex tasks (like classifying patent categories) than on simple ones (like sentiment analysis). On complex tasks, the model has to "work harder" to fit the ID data, which often makes the OOD regions even sharper and easier to distinguish.
Key Results
The authors evaluated BLOOD across a wide array of text classification tasks using RoBERTa and ELECTRA, testing against both "Far-OOD" datasets and "Background Shifts" (e.g., a model trained on short movie reviews evaluating long Yelp reviews).
- Superior Performance: $BLOOD_L$ consistently outperformed other white-box and black-box baselines (like Maximum Softmax Probability or Energy scores) across most datasets.
- Rivaling Open-Box Methods: In several cases, BLOOD performed as well as or better than "open-box" methods that actually have access to the training data.
- Background Shift Sensitivity: BLOOD proved particularly adept at detecting background shifts, outperforming even complex ensemble methods.
Limitations and Future Work
While powerful, BLOOD is not a silver bullet:
- Computational Cost: Although the estimator is efficient, it still requires multiple forward passes ($M=50$) to get a stable estimate, making it slower than simple softmax-based methods.
- Semantic Shift: The method is less effective at "semantic shift"—detecting OOD data that comes from the same domain but belongs to a different class (e.g., a news classifier seeing a new, unseen category of news).
The authors suggest that future work could explore combining BLOOD with output-based methods or investigating why specific layers (usually the last one) are so much more informative for OOD detection.
Summary updated