Improving the Behaviour of Vision Transformers with Token-consistent Stochastic Layers

Nikola Popović, Danda Pani Paudel, Thomas Probst, Luc Van Gool

Abstract

We introduce token-consistent stochastic layers in vision transformers, without causing any severe drop in performance. The added stochasticity improves network calibration, robustness and strengthens privacy. We use linear layers with token-consistent stochastic parameters inside the multilayer perceptron blocks, without altering the architecture of the transformer. The stochastic parameters are sampled from the uniform distribution, both during training and inference. The applied linear operations preserve the topological structure, formed by the set of tokens passing through the shared multilayer perceptron. This operation encourages the learning of the recognition task to rely on the topological structures of the tokens, instead of their values, which in turn offers the desired robustness and privacy of the visual features. The effectiveness of the token-consistent stochasticity is demonstrated on three different applications, namely, network calibration, adversarial robustness, and feature privacy, by boosting the performance of the respective established baselines.

1. Introduction

Vision Transformers (ViTs) have emerged as a powerful architecture for computer vision tasks. However, they inherit several vulnerabilities common to deep neural networks: overconfidence in incorrect predictions (poor calibration), susceptibility to adversarial perturbations, and the risk of privacy leaks where original inputs can be reconstructed from shared feature maps. While stochasticity is a known regularization technique, standard methods like Dropout often struggle to balance these concerns without significantly degrading model accuracy. This work proposes a novel form of randomness—token-consistent stochasticity—to address these weaknesses while maintaining high performance.

2. Token-consistent Stochastic Layers

The core innovation lies in how noise is injected into the transformer's Multi-Layer Perceptron (MLP) blocks. In a standard ViT, an image is decomposed into a set of visual tokens. The proposed method inserts a stochastic linear operation after the first fully connected layer of the MLP.

Unlike standard Dropout, which applies noise independently to every value, the Token-consistent Stochastic Layer samples a single random vector for an entire block and applies it identically to all tokens. Mathematically, this is represented as a square diagonal matrix $A$ sampled from a uniform distribution $U(1-\Delta, 1+\Delta)$. Because every token is scaled by the same factor, the transformation is a homeomorphism that preserves the topological relationships between tokens.

Comparison between the proposed token-consistent stochastic layer and standard dropout.

This consistency forces the network to become invariant to global scaling, shifting its focus from raw pixel values to the underlying structure of the image. Implementation is highly efficient, requiring no architectural changes and only a brief 30-epoch fine-tuning phase to adapt pre-trained models.

3. Experimental Results

3.1 Accuracy and Calibration

Evaluations on ImageNet-1k demonstrate that the DeiT-S model maintains robust performance even under significant noise. As shown in Figure 3, while accuracy naturally decreases as the noise intensity $\Delta$ increases, the model remains competitive. Crucially, calibration improves significantly: the Expected Calibration Error (ECE) dropped from 0.026 in the baseline to 0.010 with the stochastic layers, indicating that the model's confidence scores are far more reliable.

Impact of noise intensity (Training Δ) on ImageNet-val accuracy for different sampling strategies.

3.2 Robustness and Privacy

The method provides substantial gains in adversarial defense and data privacy:

  • Adversarial Robustness: When integrated with adversarial training, the approach achieved 49.32% accuracy under PGD-10 attacks, a notable improvement over the 42.33% baseline.
  • Feature Privacy: By introducing stochasticity during inference, the network creates a "moving target" for reconstruction attacks. The L1 loss for attackers attempting to recover original images from feature maps nearly doubled, effectively obfuscating sensitive visual details.

4. Conclusion

Token-consistent stochastic layers offer a lightweight yet effective mechanism to enhance the reliability of Vision Transformers. By prioritizing topological structure over individual feature values, the method improves calibration, robustness, and privacy with minimal impact on standard accuracy. Future work will explore the impact of this noise injection on dataset subgroups and investigate potential certifiable defense properties.

Summary updated

Abstract

We introduce token-consistent stochastic layers in vision transformers, without causing any severe drop in performance. The added stochasticity improves network calibration, robustness and strengthens privacy. We use linear layers with token-consistent stochastic parameters inside the multilayer perceptron blocks, without altering the architecture of the transformer. The stochastic parameters are sampled from the uniform distribution, both during training and inference. The applied linear operations preserve the topological structure, formed by the set of tokens passing through the shared multilayer perceptron. This operation encourages the learning of the recognition task to rely on the topological structures of the tokens, instead of their values, which in turn offers the desired robustness and privacy of the visual features. The effectiveness of the token-consistent stochasticity is demonstrated on three different applications, namely, network calibration, adversarial robustness, and feature privacy, by boosting the performance of the respective established baselines.
BibTeX
@misc{popovic2021improving,
  author = {Nikola Popović and Danda Pani Paudel and Thomas Probst and Luc Van Gool},
  title = {{Improving the Behaviour of Vision Transformers with Token-consistent Stochastic Layers}},
  year = {2021},
  eprint = {2112.15111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.15111},
}
Figure