Abstract
Current unsupervised visual representation learning methods typically employ a teacher-student framework, where the teacher is updated via an Exponential Moving Average (EMA) of the student. While effective, this approach suffers from temporal inconsistency; the teacher's "short-term memory" leads to rapidly shifting targets that introduce noise and cause catastrophic forgetting. We propose Temporal Knowledge Consistency (TKC), a framework that leverages snapshots of the model from multiple past training stages. By utilizing a Knowledge Transformer to adaptively ensemble these past states, TKC provides a more stable and consistent supervision signal. TKC achieves significant gains over state-of-the-art methods like MoCo v2 and BYOL on ImageNet and various downstream tasks.
Introduction
The "instance discrimination" paradigm has become the standard for unsupervised visual representation learning. In this setup, a student model learns by matching the representations of different augmentations of the same image produced by a teacher model. Most modern frameworks, such as MoCo and BYOL, use an EMA teacher. However, the authors identify a critical flaw: the EMA teacher only ensembles models from the very recent past. Consequently, the teacher's output for a specific image can change dramatically as training progresses, leading to temporal inconsistency. This inconsistency acts as noise, hindering the learning of robust features and causing the model to "forget" useful information learned in earlier stages.
Methods: Temporal Knowledge Consistency
To address the limitations of short-term memory in EMA teachers, the Temporal Knowledge Consistency (TKC) framework introduces a mechanism for "long-term memory" through three primary components.
Temporal Teachers and Knowledge Transformer
Instead of relying solely on the current EMA teacher ($T_n$), TKC maintains a set of "temporal teachers" ($T_{n-1}, \dots, T_{n-h}$), which are snapshots of the model from previous training stages. To avoid the high computational cost of storing multiple full models, the framework uses a History Bank to store representations (embeddings) on the CPU, moving only the required batch to the GPU during training.
Rather than using a fixed decay factor to ensemble these teachers, TKC employs a Knowledge Transformer. This small MLP dynamically learns the importance of different past states, filtering out noise and selecting the most consistent information to supervise the student.
Temporal Loss
The training objective is extended via a Temporal Loss. This loss maximizes the mutual information between the current student's representation and the targets provided by both the EMA teacher and the weighted ensemble of temporal teachers. This ensures that the model remains consistent with its own past "selves" across time.
Experimental Results
The researchers evaluated TKC on standard benchmarks, including ImageNet-1k, PASCAL VOC, and MS COCO.
ImageNet Performance
TKC consistently improves upon strong baselines. When applied to MoCo v2, it yields a 1.5% improvement in linear evaluation. More notably, when integrated with BYOL, it achieves a Top-1 accuracy of 72.4%, a 2.3% boost over the baseline.
Semi-Supervised and Downstream Tasks
The benefits of temporal consistency are most apparent in data-scarce scenarios. In semi-supervised learning with only 1% of ImageNet labels, TKC outperforms MoCo v2 by a substantial 9.6%. Furthermore, TKC demonstrates superior transferability. On PASCAL VOC object detection, it achieves 81.8% AP, surpassing even supervised pre-training (81.3%).
Discussion and Conclusion
TKC highlights that temporal consistency—ensuring a model's understanding of an image remains stable over time—is as vital as spatial consistency for self-supervised learning. By adaptively ensembling historical knowledge, TKC mitigates the noise inherent in EMA-based frameworks. While the approach introduces some computational overhead (approximately 47% increase in training time per epoch), the significant gains in accuracy and robustness, particularly in semi-supervised settings, make it a compelling advancement for unsupervised representation learning. Future work may explore more sophisticated attention mechanisms within the Knowledge Transformer to further refine the temporal signal.
Summary updated