Abstract
We introduce token-consistent stochastic layers in vision transformers, without causing any severe drop in performance. The added stochasticity improves network calibration, robustness and strengthens privacy. We use linear layers with token-consistent stochastic parameters inside the multilayer perceptron blocks, without altering the architecture of the transformer. The stochastic parameters are sampled from the uniform distribution, both during training and inference. The applied linear operations preserve the topological structure, formed by the set of tokens passing through the shared multilayer perceptron. This operation encourages the learning of the recognition task to rely on the topological structures of the tokens, instead of their values, which in turn offers the desired robustness and privacy of the visual features. The effectiveness of the token-consistent stochasticity is demonstrated on three different applications, namely, network calibration, adversarial robustness, and feature privacy, by boosting the performance of the respective established baselines.
Introduction
Vision Transformers (ViTs) have emerged as a powerful architecture for computer vision tasks. However, like many deep neural networks, they suffer from common weaknesses: poor calibration (overconfidence), vulnerability to adversarial attacks, and privacy risks regarding feature reconstruction in collaborative environments. Standard regularization techniques like Dropout introduce randomness but often fail to respect the inherent "set-of-tokens" structure of Transformers. This research introduces "token-consistent" stochasticity to address these weaknesses by preserving the topological relationships between tokens while injecting noise.
Methods: Token-Consistent Stochastic Layers
The core innovation is the introduction of Token-consistent Stochastic Layers. Unlike standard Dropout, which independently zeros out neurons, this method applies a stochastic linear operation within the Multi-Layer Perceptron (MLP) blocks.
As shown in Figure 2, the mechanism involves sampling a random vector from a uniform distribution $U(1-\Delta, 1+\Delta)$. This vector forms a diagonal matrix $A$ that scales the feature vectors of all tokens in a block identically. This "token-consistency" ensures that the relative "shape" or topology of the token set is preserved (a homeomorphism), forcing the model to learn structural relationships rather than memorizing raw feature values. Implementation is straightforward, requiring only a brief fine-tuning period (e.g., 30 epochs on ImageNet-1k) without architectural changes.
Experimental Results
The primary evaluation was conducted on ImageNet-1k, with further testing on 12 transfer learning datasets including CIFAR, Caltech, and Oxford Flowers.
Performance and Calibration
A key finding is that ViTs are remarkably robust to this noise. As seen in Figure 3, even with significant noise ($\Delta = 1.0$), accuracy remains high when using Monte Carlo (MC) sampling during inference. For DeiT-S, accuracy matched the baseline at ~79.6%. More importantly, the Expected Calibration Error (ECE) improved dramatically, dropping from 0.088 to as low as 0.010, indicating much more reliable confidence scores.
Robustness and Privacy
The method significantly enhances adversarial defense. Under a PGD-10 attack, a standard DeiT-S model's accuracy collapses, whereas the stochastic version retains 49.32% accuracy. In terms of privacy, the token-consistent noise makes it substantially harder for reconstruction attacks to recover original images from intermediate features, as evidenced by higher L1/L2 errors and lower visual quality (PSNR/SSIM) compared to standard Dropout.
Conclusion
Token-consistent stochastic layers provide a robust framework for improving ViT calibration, security, and privacy without sacrificing performance. While Monte Carlo sampling increases inference time and fine-tuning is required, the benefits in reliability and robustness are significant. Future work may explore more complex homeomorphic mappings to further enhance these properties.
Summary updated