Constraining Representations Yields Models That Know What They Don't Know

João Monteiro, Pau Rodríguez, Pierre-André Noël, Issam H. Laradji, David Vázquez

Introduction

Neural networks are notoriously overconfident, often assigning high probability to incorrect or nonsensical predictions. This "silent failure" of deep learning is a major hurdle for deployment in safety-critical systems. When a standard neural network encounters an input it doesn't understand—such as a blurred image or an out-of-distribution category—it frequently produces a high-confidence, yet completely wrong, prediction.

Most existing solutions rely solely on the final output layer (the "logits") to estimate confidence. However, by the time an error reaches the final layer, it is often too late to catch. Standard models have too much freedom in their intermediate representations, making them easy to "fool" and difficult to audit for uncertainty. To address this lack of internal structural constraints, researchers introduce Total Activation Classifiers (TAC). TAC forces a model’s internal activations across multiple layers to match pre-defined, random binary "class codes," generating a reliable confidence score based on how far the internal state deviates from these codes.

Methods: Total Activation Classifiers (TAC)

The core idea behind TAC is to turn classification into a pattern-matching task across the entire depth of the network:

  1. Class Codes: Each class is assigned a unique, fixed, and randomly generated binary vector (a "code") at the start of training. These codes are designed to be approximately orthogonal (highly distinct from one another).
  2. Activation Profiles: Instead of just looking at the final layer, TAC "slices" the activations from multiple intermediate layers. These slices are reduced (via summing or averaging) and concatenated to form an "activation profile"—a snapshot of the model's internal state.
Architecture of Total Activation Classifiers (TAC) matching layer activations to class codes.
  1. Constrained Training: The model is trained using a hybrid loss function. It must not only predict the correct class but also ensure its internal activation profile matches the target class code.
  2. Inference & Rejection: At test time, the model calculates the distance (e.g., $L_1$ or Cosine distance) between its current activation profile and the valid class codes. If the profile is too far from any valid code, the model can "reject" the input or defer to a human, effectively saying, "I don't know."

TAC can be trained from scratch or added as a lightweight "add-on" to frozen, pre-trained models like BERT or Vision Transformers (ViT) using small projection layers.

Datasets

The researchers evaluated TAC across diverse modalities:

  • Computer Vision: MNIST (handwritten digits), CIFAR-10 (small images), and ImageNet (large-scale image recognition).
  • Natural Language Processing: Intent classification datasets including HWU64, Banking77, and CLINC150. These were chosen specifically because they involve many classes and frequent "out-of-scope" queries.

Key Insights

  • Enforcing vs. Hoping: Previous research noted that class-dependent patterns sometimes emerge naturally. TAC moves from "hoping" these patterns appear to "enforcing" them as a structural requirement.
  • The "Security Guard" Effect: By tying labels to internal states, TAC makes life much harder for adversarial attackers. To fool the model, an attacker can't just flip the final output; they must manipulate activations across the entire depth of the network to match a specific, secret binary code.
  • Zero-Cost Accuracy: Adding TAC as an add-on to pre-trained models does not degrade their original accuracy. It simply provides an additional, reliable check on internal consistency.
  • Orthogonality Matters: Using high-dimensional random codes ensures that classes are far apart in the representation space, providing a "buffer" that helps detect subtle errors.

Key Results

  • Superior Error Detection: In "rejection" scenarios (where a model is allowed to abstain), TAC consistently outperformed standard metrics like Maximum Softmax Probability (MSP). On ImageNet, TAC significantly improved the "Value" of the classifier across a wide range of error costs.
  • Out-of-Distribution (OOD) Success: On the CLINC150 intent detection benchmark, TAC achieved a 97.28% AUROC for detecting unseen classes, outperforming state-of-the-art baselines.
  • Adversarial Robustness: In a "monitored" setting (where the TAC module is kept secret from the attacker), it raised the robust accuracy of a base model from near 0% to over 53% against strong PGD attacks, while maintaining high clean accuracy (over 90%).
Histogram of L1 distances separating clean images from adversarial attacks.
  • Efficiency: Training the TAC add-on for text models takes only a few hours on a single GPU, making it highly practical for real-world engineering workflows.

Limitations and Future Work

While highly effective, TAC has a few limitations. If an attacker has full "white-box" access to the TAC codes and projection layers, they could potentially craft attacks that satisfy the TAC constraints, though this remains significantly more complex than attacking a standard head. Additionally, choosing which layers to monitor and which distance metric to use currently requires data-dependent hyperparameter tuning. Finally, training with the binary matching loss alone can be unstable, requiring a hybrid loss to ensure smooth convergence.

Future work may involve exploring deterministic code generation (like Hadamard matrices) or automatically learning the optimal "slicing" strategy for different architectures.

Summary updated

Abstract

A well-known failure mode of neural networks is that they may confidently return erroneous predictions. Such unsafe behaviour is particularly frequent when the use case slightly differs from the training context, and/or in the presence of an adversary. This work presents a novel direction to address these issues in a broad, general manner: imposing class-aware constraints on a model's internal activation patterns. Specifically, we assign to each class a unique, fixed, randomly-generated binary vector - hereafter called class code - and train the model so that its cross-depths activation patterns predict the appropriate class code according to the input sample's class. The resulting predictors are dubbed total activation classifiers (TAC), and TACs may either be trained from scratch, or used with negligible cost as a thin add-on on top of a frozen, pre-trained neural network. The distance between a TAC's activation pattern and the closest valid code acts as an additional confidence score, besides the default unTAC'ed prediction head's. In the add-on case, the original neural network's inference head is completely unaffected (so its accuracy remains the same) but we now have the option to use TAC's own confidence and prediction when determining which course of action to take in an hypothetical production workflow. In particular, we show that TAC strictly improves the value derived from models allowed to reject/defer. We provide further empirical evidence that TAC works well on multiple types of architectures and data modalities and that it is at least as good as state-of-the-art alternative confidence scores derived from existing models.
BibTeX
@inproceedings{monteiro2023constraining,
  author = {João Monteiro and Pau Rodríguez and Pierre-André Noël and Issam H. Laradji and David Vázquez},
  title = {{Constraining Representations Yields Models That Know What They Don't Know}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
}
Figure