Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothness

Fran Jelenic, Josip Jukic, Martin Tutek, Mate Puljiz, Jan Snajder

Abstract

Modern machine learning models often exhibit overconfidence when encountering out-of-distribution (OOD) data, assigning high-probability labels to unfamiliar inputs. We present BLOOD (Between-Layer Out-Of-Distribution Detection), a novel "white-box" method that identifies OOD samples by measuring the "smoothness" of internal representation transformations. By analyzing the Jacobian matrix between layers, we demonstrate that fine-tuned models develop smooth paths for in-distribution data, while OOD data triggers sharper, less predictable transitions. BLOOD provides a robust detection mechanism without requiring access to original training data.

Introduction

The challenge of overconfidence in neural networks remains a significant hurdle for reliable AI deployment. When presented with data outside their training distribution, models frequently fail to signal uncertainty. Existing detection methods are typically categorized as "black-box," relying on often unreliable output scores, or "open-box," requiring access to the original training sets. BLOOD offers a "white-box" alternative, utilizing only the model's internal weights and architecture to detect OOD samples by identifying intrinsic properties of the model's internal processing.

Methodology

BLOOD quantifies the "smoothness" of the transformation between consecutive layers $l$ and $l+1$. This is defined using the Frobenius norm of the Jacobian matrix; essentially, if a small change in a layer's input leads to a small, predictable change in its output, the transformation is considered "smooth."

To handle the high dimensionality of modern Transformers, the authors employ an efficient unbiased estimator using Jacobian-vector products via forward-mode automatic differentiation. Two primary variants are explored: $BLOOD_M$, which averages smoothness across all layers, and $BLOOD_L$, which focuses solely on the final layer's transformation before the output.

Figure 1: Comparison of the smoothness measure across layers for Train, In-Distribution (ID), and Out-of-Distribution (OOD) data.

Key Insights

The efficacy of BLOOD stems from the observation that the fine-tuning process essentially "carves out" a smooth path through the representation space for in-distribution (ID) data. As a model learns a specific task, it organizes its internal abstractions to allow for seamless transitions. OOD data, lacking this learned structure, results in "jagged" or sharp transitions between layers, as shown in the higher smoothness measures for OOD samples across the network.

Last Layer Dominance

A critical finding is the dominance of the final layers in OOD detection. The upper layers of a Transformer are the most specialized and undergo the most radical "smoothing" during fine-tuning. Consequently, the disparity in transformation dynamics between ID and OOD data is most pronounced at the end of the network, making $BLOOD_L$ a particularly potent detector.

Figure 3: Representation change across Transformer layers, highlighting the significant shift in the final layer.

Results and Performance

Evaluated on text classification (RoBERTa, ELECTRA) and image recognition (ViT, CNN), BLOOD consistently outperforms standard baselines like Maximum Softmax Probability (MSP) and energy-based scores.

  • Superior Accuracy: On the TREC dataset using ELECTRA, $BLOOD_L$ achieved an AUROC of 98.90%.
  • Domain Robustness: The method is particularly effective at detecting "background shifts"—changes in the underlying domain or style of the text—improving OOD detection by nearly 20 percentage points over traditional methods on complex datasets like BigPatent.
  • Complexity Advantage: BLOOD performs significantly better on complex tasks where the model must work harder to organize the ID representation space, making the "unorganized" nature of OOD data more apparent.

Conclusion and Future Work

BLOOD provides a powerful, data-free method for OOD detection by tapping into the internal geometric properties of model transformations. While it faces challenges with "Near-OOD" semantic shifts (where domain remains similar but classes change) and introduces some computational overhead due to Jacobian calculations, its ability to detect domain shifts through internal "smoothness" marks a significant step forward in building more self-aware and reliable machine learning systems.

Summary updated

BibTeX
@inproceedings{jelenic2024out,
  author = {Fran Jelenic and Josip Jukic and Martin Tutek and Mate Puljiz and Jan Snajder},
  title = {{Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothness}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=AcRfzLS6se},
}
Figure