Abstract
The evolution of large-scale contrastive pre-training propelled by top-tier datasets has reached a transition point in the scaling law. Consequently, sustaining and enhancing a model's pre-training capabilities in drift environments have surfaced as a notable challenge. In this paper, we initially uncover that contrastive pre-training methods are significantly impacted by concept drift wherein distributions change unpredictably, resulting in notable biases in the feature space of the pre-trained model. Empowered by causal inference, we construct a structural causal graph to analyze the impact of concept drift to contrastive pre-training systemically, and propose the causal interventional contrastive objective. Upon achieving this, we devise a resilient contrastive pre-training approach to accommodate the data stream of concept drift, with simple and scalable implementation. Extensive experiments on various downstream tasks demonstrate our resilient contrastive pre-training effectively mitigates the bias stemming from the concept drift data stream. Codes are available at https://anonymous.4open.science/r/ResilientCL/.
Introduction: The Challenge of Concept Drift
Standard vision models are typically trained on static, "frozen" datasets. However, real-world data streams—such as those from autonomous vehicle fleets or medical imaging networks—are dynamic and subject to concept drift. This phenomenon involves unpredictable changes in the underlying data distribution over time, such as shifting class frequencies or varying environmental conditions.
Current state-of-the-art contrastive learning (CL) frameworks, like MoCo or DINO, utilize a "student-teacher" architecture where the teacher is an Exponential Moving Average (EMA) of the student's parameters. While effective for static data, this mechanism introduces a "stale bias" in non-stationary environments. The teacher retains outdated information from previous distributions, providing skewed supervision that leads to a collapsed feature space. In these scenarios, rare "tail" categories are often overwhelmed by dominant "head" categories.
Methodology: Causal Intervention for De-biasing
To address this, the authors propose Resilient Contrastive Pre-training (RCP), grounded in causal inference. By modeling concept drift as a "confounder" in a Structural Causal Model (SCM), they identify how it creates spurious correlations between features and predictions.
Resilient Contrastive Pre-training (RCP)
The RCP framework introduces three key components to "de-confound" the learning process:
- Causal Intervention: RCP applies a "backdoor adjustment" ($P(Y|do(X))$), forcing the model to consider all potential drift contexts equally and severing the influence of the latent confounder.
- Drift Intervention Module: This module implements the causal logic using a structure similar to self-attention. It treats student features as "queries" and momentum teacher outputs as "keys" to identify and subtract latent drift bias.
- Scalable Implementation: RCP is designed as a plug-and-play enhancement for existing frameworks like MoCo v3, maintaining efficiency for large-scale Vision Transformers (ViTs).
Experimental Results
The researchers evaluated RCP across long-tailed streams (ImageNet-LT, iNaturalist 2018), domain shifts (ImageNet-V2, -Sketch, -R), and out-of-distribution (OOD) detection tasks.
- Long-Tailed Performance: RCP achieved a Top-1 accuracy of 50.3% on ImageNet-LT, outperforming MoCo v3 (48.5%) and DINO (43.3%).
- Few-Shot Robustness: The most significant gains occurred in rare categories. For classes with fewer than 20 samples, RCP reached 16.0% accuracy, while competitors struggled to exceed 12%.
- Feature Quality: In linear probing experiments, RCP's backbone reached 32.8% on ImageNet-LT, a nearly 6% improvement over MoCo v3, indicating superior representation learning.
Discussion and Future Work
RCP demonstrates that the EMA teacher, while stabilizing for static data, acts as a bottleneck in dynamic environments by preserving stale biases. By framing drift as a causal confounding problem, RCP provides a principled de-biasing method without requiring ground-truth labels of the drift.
Future research may address the sensitivity of the "drift adaptation window" and explore lighter-weight causal approximations to further reduce computational overhead. Additionally, explicitly modeling specific types of drift (e.g., semantic vs. sensor-based) could enhance the model's granularity.
Summary updated