Improving Neural Network Accuracy by Concurrently Training with a Twin Network

Benjamin Vandersmissen, Lucas Deckers, Jose OramasICLR 2025 · OpenReview

Deep learning practitioners constantly seek methods to enhance model performance without increasing inference costs. A recent study, "Improving Neural Network Accuracy by Concurrently Training with a Twin Network," adapts a technique called Twin Network Augmentation (TNA)—originally developed for Spiking Neural Networks (SNNs)—to standard Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs).

TNA trains two identical networks concurrently, matching their raw logits via a Mean Squared Error (MSE) loss. Post-training, the "twin" network is discarded, yielding a significantly more accurate model with zero added inference overhead.

The Problem: Limitations of Standard Training

Standard neural network training typically involves a single model minimizing a loss function (like Cross-Entropy) against ground-truth labels. While effective, this approach can lead to overfitting or the learning of brittle features that fail to generalize.

Existing alternatives have notable drawbacks:

  • Knowledge Distillation (KD): Requires a large, computationally expensive pre-trained "teacher" model to guide a smaller "student."
  • Deep Mutual Learning (DML): Trains a cohort of models collaboratively by matching their output probability distributions. However, matching probabilities after a SoftMax layer discards valuable, nuanced information.

TNA addresses these issues by matching the raw, unscaled logits directly, providing a richer and stronger signal for the networks to learn robust, high-quality features.

Methodology: Twin Network Augmentation

The core architecture of TNA is straightforward yet highly effective. Two networks (a "base" and a "twin") with identical architectures but different random initializations are trained simultaneously.

Comparison of training methodologies including TNA, DML, Ensembling, and KD.

During training, the total loss is the sum of three components:

  1. Base Loss: Cross-entropy for the base network.
  2. Twin Loss: Cross-entropy for the twin network.
  3. Logit Matching Loss: An MSE loss that penalizes the difference between the raw output logits of both networks.

Because the two networks start from different random initializations, they explore distinct trajectories in the feature space. Forcing their outputs to align provides mutual positive reinforcement. Crucially, at inference time, the twin is discarded, meaning the final deployed model retains the exact same computational footprint as a traditionally trained network.

Experimental Setup and Key Insights

The authors validated TNA across a rigorous suite of benchmarks, including CIFAR-10, CIFAR-100, TinyImageNet, ImageNet, and TinyImageNet-C (to test robustness against common image corruptions).

Through these experiments, several key insights emerged:

  • Logits > Probabilities: TNA consistently outperforms Deep Mutual Learning (DML). Matching raw logits preserves the predictive nuances that the SoftMax function tends to squash.
  • Efficiency Beyond Parameter Count: The performance boost is not merely a byproduct of increased training parameters. A standard ResNet-18 trained with TNA outperforms a "Wide" ResNet-18 that possesses significantly more parameters.
  • The "Same View" Requirement: TNA requires both twins to process the exact same augmented version of an image. Providing different augmentations to each twin—a common trick in contrastive learning—actually degrades performance.
  • Enhanced Deep Features: Using linear probes to test intermediate layers, researchers found that TNA specifically improves the quality of feature representations in the deeper layers of the network.

Key Results

The performance gains achieved by TNA are substantial across various architectures and datasets:

  • TinyImageNet (ResNet-18): Accuracy jumped from 59.33% to 64.84% (+5.51%).
  • CIFAR-100 (MobileNetv2): Accuracy increased from 76.17% to 79.50% (+3.33%).
  • ImageNet (ViT-S/16): Vision Transformers saw a nearly 2% boost in accuracy.
  • Robustness (TinyImageNet-C): Models trained with TNA were significantly more resilient to data corruption, averaging 32.49% accuracy compared to 29.88% for standard models.

Limitations and Future Directions

While TNA offers a "free lunch" at inference, it is computationally expensive during training. It effectively doubles the required VRAM and training time because backpropagation must occur through two networks instead of one. Additionally, TNA works best when the two networks are of similar capacity; pairing a very large twin with a small base network can cause an "averaging" effect that pulls down the performance of the stronger model.

Future research will explore "intermediate supervision"—matching features at earlier layers of the network rather than just the final output—to determine if even greater gains can be extracted from this twin-training paradigm.

Summary updated

Abstract

Recently within Spiking Neural Networks, a method called Twin Network Augmentation (TNA) has been introduced. This technique claims to improve the validation accuracy of a Spiking Neural Network simply by training two networks in conjunction and matching the logits via the Mean Squared Error loss. In this paper, we validate the viability of this method on a wide range of popular Convolutional Neural Network (CNN) benchmarks and compare this approach to existing Knowledge Distillation schemes. Next, we conduct a in-depth study of the different components that make up TNA and determine that its effectiveness is not solely situated in an increase of trainable parameters, but rather the effect of the training methodology. Finally, we analyse the representations learned by networks trained with TNA and highlight their superiority in a number of tasks, thus proving empirically the applicability of Twin Network Augmentation on CNN models.
BibTeX
@inproceedings{vandersmissen2025improving,
  author = {Benjamin Vandersmissen and Lucas Deckers and Jose Oramas},
  title = {{Improving Neural Network Accuracy by Concurrently Training with a Twin Network}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=TEmE9PSC65},
}
Figure