Abstract
Deep neural network training is inherently stochastic, often leading to inconsistent optimization trajectories and overfitting to batch-specific noise. We present Deep Companion Learning (DCL), a training framework designed to stabilize this process by enforcing "historical consistency." By utilizing a secondary companion model to forecast the primary model's predictions based on its historical trajectory, DCL provides a stable semantic signal that guides the model toward more robust representations. Our experiments demonstrate significant accuracy improvements across various benchmarks, including a 4.1% gain on CIFAR-100, and show that DCL can even allow models trained from scratch to outperform those initialized with large-scale pre-training.
1. Introduction
The standard paradigm for training deep neural networks relies on Stochastic Gradient Descent (SGD). While effective, the stochastic nature of SGD introduces significant variability into the learning process. This variability manifests in two primary ways: batch variability, where gradients from small data subsets do not represent the global loss surface, and trajectory variability, where the model's path through parameter space becomes jittery and inconsistent. This "forgetfulness"—the tendency of a model to diverge from previously learned semantic patterns when exposed to new random batches—hinders the model's ability to generalize to unseen data.
2. The Deep Companion Learning Framework
To address the instability of SGD trajectories, Deep Companion Learning (DCL) introduces a parallel training architecture. The framework consists of a primary model (parameterized by $\theta$) and a Deep Companion Model (DCM, parameterized by $\omega$).
2.1 Historical Forecasting in Logit Space
Unlike existing methods that average model parameters (such as Stochastic Weight Averaging), DCL operates in the logit space. The companion model maintains a running consensus of the primary model's historical states to provide a "forecast" of what the logits for a given input should be. This forecast acts as a stable reference point that filters out the noise of individual SGD steps.
2.2 Dynamic Regularization
The training objective is formulated as a multi-task optimization problem. The primary model is trained to minimize a standard cross-entropy loss while simultaneously minimizing the distance (typically Mean Squared Error) between its current predictions and the companion's historical forecast. This dual objective ensures that the model remains consistent with its own learned history while adapting to new data.
2.3 Computational Efficiency
To mitigate the overhead of training two models, the authors propose several efficient variants:
- Deep Companion Prototypes (DCP): Utilizing class-based prototypes instead of a full secondary network.
- Asymmetric Architectures: Using a significantly smaller companion model (e.g., a ResNet-10) to guide a larger primary model (e.g., a ResNet-18), which maintains performance gains while reducing FLOPs.
3. Experimental Evaluation
The efficacy of DCL was validated across a diverse suite of computer vision benchmarks, ranging from small-scale datasets to large-scale industrial standards.
3.1 Benchmarks and Performance
- CIFAR-100: A ResNet-18 architecture saw an improvement from 78.3% to 82.4% accuracy.
- ImageNet-1K: DCL provided a ~1% boost to ResNet-50 and demonstrated compatibility with Transformer-based architectures like ViT-T and Swin-T.
- Downstream Tasks: Models trained with DCL showed superior transferability when fine-tuned on specialized datasets such as CUB-200, Oxford-Pet, and Food-101.
3.2 Versatility Across Paradigms
DCL is not limited to supervised learning. It has shown to be a "plug-and-play" enhancement for:
- Semi-supervised learning: Improving FixMatch performance by 1-3%.
- Self-supervised learning: Enhancing the representations learned by Masked Autoencoders (MAE).
- Knowledge Distillation: Enabling student models to achieve higher accuracy even in the absence of a pre-trained teacher.
4. Key Insights
4.1 Latent Semantic Structure
A critical finding of the DCL framework is its ability to capture semantic relationships within the model's "errors." For instance, when classifying a "Leopard," the companion model consistently maintains high logit values for semantically related classes like "Tiger" and "Lion." Standard training often produces noisy, irrelevant class rankings. By forcing the primary model to adhere to the companion's structured confusion, DCL encourages the development of more logical and linearly separable feature spaces.
4.2 Reducing Non-Target Perplexity
DCL significantly reduces "non-target perplexity," meaning the model becomes more certain about which classes a sample does not belong to. This clarity in the feature space leads to better clustering and more robust decision boundaries.
4.3 Training from Scratch vs. Pre-training
Notably, a ResNet-18 trained from scratch using DCL on CIFAR-100 outperformed a standard ResNet-18 that had been pre-trained on ImageNet. This suggests that optimizing the training dynamics through historical consistency can be as effective as—or even superior to—leveraging massive external datasets.
5. Limitations and Future Directions
Despite its performance benefits, DCL introduces additional computational costs during the training phase, effectively doubling memory requirements in its standard form. While the Prototype and smaller companion variants alleviate this, a trade-off remains. Furthermore, the framework relies on a smoothing hyperparameter ($\alpha$) that requires careful tuning to balance historical memory with current updates. Future research will explore the application of historical consistency in domains beyond computer vision, such as Natural Language Processing and Reinforcement Learning.
6. Conclusion
Deep Companion Learning offers a robust solution to the inherent instability of SGD-based training. By enforcing consistency with a model's own historical trajectory, DCL captures latent semantic structures that standard training misses, leading to state-of-the-art generalization across a wide array of tasks and architectures.
Summary updated