Data-Efficient Double-Win Lottery Tickets from Robust Pre-training

Tianlong Chen, Zhenyu Zhang, Sijia Liu, Yang Zhang, Shiyu Chang, Zhangyang Wang

Abstract

This paper introduces Double-Win Lottery Tickets: extremely sparse subnetworks that maintain both high standard accuracy and adversarial robustness when transferred to new tasks. The researchers demonstrate that robust pre-training is the critical component for identifying these tickets. These subnetworks are not only more resilient to adversarial perturbations but also significantly more data-efficient than their standard counterparts, particularly when training data for downstream tasks is scarce.

Introduction

Modern deep learning typically follows a pre-train and fine-tune paradigm using massive models. While the Lottery Ticket Hypothesis (LTH) suggests that these large models contain tiny, sparse subnetworks capable of matching dense performance, most LTH research has focused exclusively on standard accuracy. In real-world applications, security is a primary concern; models are often vulnerable to adversarial attacks—small, intentional input perturbations that cause incorrect predictions.

Standard "winning tickets" often fail to maintain robustness during transfer. This paper addresses this gap by asking: Can we identify sparse subnetworks that "win" twice—matching dense model performance in both standard accuracy and adversarial robustness across diverse training regimes?

Methodology

The authors propose a new paradigm for identifying "Double-Win" tickets, focusing on the influence of pre-trained weights over the pruning process itself.

Overview of the Double-Win Lottery Ticket paradigm, showing the identification of tickets from robust pre-training and their transfer to downstream tasks.

The process involves four key stages:

  1. Robust Pre-training: Instead of standard pre-training, the authors employ Adversarial Training (AT) and Fast Adversarial Training (FAT) on ImageNet to establish a robust weight initialization.
  2. Iterative Magnitude Pruning (IMP): The standard IMP algorithm is used to identify sparse masks. A key finding is that the choice of pre-trained weights is the primary driver of success, rather than the use of adversarial objectives during the pruning phase.
  3. Transfer Learning: The identified sparse subnetworks are transferred to downstream tasks (e.g., CIFAR-10, CIFAR-100, SVHN) using both standard and adversarial fine-tuning.
  4. Data-Limited Evaluation: To assess practical utility, transfer performance is evaluated in data-scarce environments, specifically using only 1% or 10% of the available downstream training data.

Results and Analysis

The experimental results reveal several critical insights into the nature of robustly pre-trained subnetworks across various benchmarks, including CUB-200, VisDA17, and VOC.

Robustness and Sparsity

Robust pre-training facilitates the discovery of extremely sparse tickets. For instance, on CIFAR-10, the authors identified double-win tickets at 91.41% sparsity using adversarial pre-training, outperforming the ~89% sparsity limit of standard pre-training. Subnetworks derived from AT pre-training consistently surpassed FAT and standard counterparts in robust accuracy (RA) across all benchmarks.

The Flatter Landscape Advantage

A central discovery is that robust pre-training leads to "flatter" loss basins. Flatness in the loss landscape is a known indicator of superior generalization, which explains why these tickets perform exceptionally well when data is limited.

Loss landscape visualizations showing that robust pre-training leads to flatter basins, which aids generalization in data-scarce settings.

Data Efficiency Gains

The most significant practical advantage of Double-Win tickets is their data efficiency. In scenarios with only 1% of the target dataset, these tickets drastically outperform standard subnetworks. For CIFAR-10, robustly pre-trained tickets improved standard accuracy by up to 30.97% and robust accuracy by up to 8.36% compared to standard tickets.

Conclusion and Future Work

Double-Win tickets offer a robust and efficient alternative for deploying models in resource-constrained or security-sensitive environments. While the initial computational cost of robust pre-training and IMP is high, the resulting subnetworks provide substantial gains in data efficiency and resilience. Future research will explore the existence of these properties in other domains, such as Natural Language Processing, and investigate methods to reduce the initial search cost.

Summary updated

BibTeX
@inproceedings{chen2022data,
  author = {Tianlong Chen and Zhenyu Zhang and Sijia Liu and Yang Zhang and Shiyu Chang and Zhangyang Wang},
  title = {{Data-Efficient Double-Win Lottery Tickets from Robust Pre-training}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=BliyjqWlkqv},
}
Figure