Abstract
The "pre-train and fine-tune" paradigm has become a cornerstone of modern machine learning, yet it introduces significant security vulnerabilities. Publicly available self-supervised learning (SSL) encoders are susceptible to Downstream-Agnostic Adversarial Examples (DAEs)—universal perturbations that can compromise any downstream model inheriting the encoder's features. This paper presents Genetic Evolution-Nurtured Adversarial Fine-tuning (Gen-AF), a novel two-stage framework designed to secure encoders without sacrificing performance. By combining genetic-inspired regularization with selective layer evolution, Gen-AF maintains the "knowledge" of pre-trained models while providing robust defense against state-of-the-art adversarial attacks.
The Vulnerability of Pre-trained Encoders
In the current AI landscape, large-scale encoders are often released by major providers for public use. While efficient, this transparency allows attackers to craft adversarial examples that target the encoder's feature extraction process directly. Because these features are fundamental to any downstream task—whether it be medical diagnosis or autonomous navigation—the vulnerability is inherited by all subsequent models.
Traditional defenses often struggle with the "pre-training dilemma." Standard adversarial training can lead to domain shift, where the defense is less effective on downstream data, and catastrophic forgetting, where the model loses the high-quality feature extraction capabilities (its "genes") it acquired during pre-training.
The Gen-AF Framework
To address these challenges, researchers developed Gen-AF, which operates in two distinct evolutionary stages to balance robustness and accuracy.
Stage I: Genetic-Driven Dual-Track Adversarial Fine-tuning (GDAF)
The first stage aims to build a foundation of robustness while preserving the encoder's core "genes." It utilizes a bilevel optimizer that applies a very low learning rate to the encoder to prevent drastic parameter shifts. Crucially, it introduces Genetic Regularization, which focuses on preserving the topology of the feature space. Instead of aligning individual data points, the model ensures that the relative relationships between samples are maintained even under adversarial pressure.
Stage II: Evolutionary Adaptability Fine-tuning (EAT)
The second stage focuses on recovering clean-image accuracy. The framework performs a Robustness Assessment to identify "robust-redundant" layers—those that contribute significantly to accuracy but minimally to adversarial defense. By freezing the robust layers and selectively fine-tuning only the redundant ones using clean data, the model achieves an optimal balance between security and performance.
Experimental Results and Key Insights
The effectiveness of Gen-AF was evaluated across ten SSL methods (including SimCLR, MoCo, and DINO) and six diverse datasets such as CIFAR10, GTSRB, and ImageNet20.
- State-of-the-Art Robustness: Gen-AF demonstrated exceptional resilience against potent attacks like AdvEncoder. While standard adversarial training methods often saw Robust Testing Accuracy (RA) drop below 20%, Gen-AF consistently maintained RA levels between 62% and 66%.
- Performance Generalization: Unlike many defensive techniques that degrade clean-image performance, Gen-AF often improved Clean Testing Accuracy (TA). For instance, a SimCLR model on the STL10 dataset saw its TA increase from 62.81% to 68.69%.
- Backdoor Immunity: Beyond standard adversarial defense, Gen-AF proved capable of "purifying" models infected with backdoors (e.g., BadEncoder), reducing attack success rates from nearly 100% to as low as 3.24%.
- Computational Efficiency: The framework is highly efficient, requiring approximately one-third of the training time per epoch compared to traditional adversarial training methods like TRADES.
Conclusion
Gen-AF represents a significant step forward in securing the "pre-train and fine-tune" pipeline. By treating model parameters as evolving "genes" and prioritizing the preservation of feature-space topology, it successfully overcomes the trade-off between robustness and accuracy. Future research may extend these evolutionary principles to larger foundation models and other domains such as image classification and beyond.
Summary updated