Abstract
This paper introduces RDA, a pioneering approach designed to address two primary deficiencies prevalent in previous endeavors aiming at stealing pre-trained encoders: (1) suboptimal performances attributed to biased optimization objectives, and (2) elevated query costs stemming from the end-to-end paradigm that necessitates querying the target encoder every epoch. Specifically, we initially Refine the representations of the target encoder for each training sample, thereby establishing a less biased optimization objective before the steal-training phase. This is accomplished via a sample-wise prototype, which consolidates the target encoder's representations for a given sample's various perspectives. Demanding exponentially fewer queries compared to the end-to-end approach, prototypes can be instantiated to guide subsequent query-free training. For more potent efficacy, we develop a multi-relational extraction loss that trains the surrogate encoder to Discriminate mismatched embedding-prototype pairs while Aligning those matched ones in terms of both amplitude and angle. In this way, the trained surrogate encoder achieves state-of-the-art results across the board in various downstream datasets with limited queries. Moreover, RDA is shown to be robust to multiple widely-used defenses.
Introduction
High-performance self-supervised learning (SSL) encoders, such as OpenAI's CLIP or Google's ImageNet models, are valuable assets requiring significant computational resources and massive datasets. To monetize these models, providers often offer API access where users pay for embeddings. However, this accessibility facilitates model stealing attacks, where an adversary attempts to train a "surrogate" encoder that replicates the target's functionality.
Existing stealing methods suffer from two major drawbacks. First, they often rely on biased objectives, where a single embedding from the target model is treated as ground truth for a given sample, ignoring the noise introduced by data augmentations. Second, "end-to-end" methods require querying the target encoder in every training epoch, leading to prohibitive query costs. This paper introduces the RDA framework (Refine, Discriminate, and Align) to overcome these limitations.
Methodology: The RDA Framework
The RDA framework operates in two distinct phases: a query phase for refinement and a query-free training phase.
1. Refine: Sample-Wise Prototypes
To mitigate the bias of single-embedding targets, RDA introduces sample-wise prototypes. For each image in the surrogate dataset, multiple augmentations (e.g., different crops or flips) are generated. These are passed through the target encoder, and the resulting embeddings are averaged to form a stable "prototype." This prototype serves as a refined, less noisy representation of the sample's core features.
2. Discriminate and Align: Multi-Relational Extraction
Once prototypes are generated and stored, the surrogate encoder is trained without further queries to the target model. The training employs a multi-relational extraction loss:
- Discriminate: A contrastive objective ensures the surrogate's embeddings are close to their corresponding prototypes and far from mismatched ones.
- Align: The surrogate is trained to match the prototype's Amplitude (magnitude) and Angle (direction). While many methods focus only on cosine similarity (angle), RDA demonstrates that matching the vector magnitude is crucial for high-fidelity replication.
Experimental Evaluation
The authors evaluated RDA against several target encoders (pre-trained on CIFAR10, STL10, ImageNet, and CLIP) using surrogate datasets like Tiny ImageNet. Performance was measured across seven downstream classification tasks, including MNIST, CIFAR100, and SVHN.
Efficiency and Performance
RDA demonstrates a significant leap in query efficiency. Compared to the previous state-of-the-art method, Cont-Steal, RDA achieved a 1.22% higher average accuracy while using only 1% of the query budget (2,500 vs. 250,000 queries). When the budget was increased to 10%, the performance gap grew to 5.20%.
Robustness and Functionality Recovery
RDA is remarkably effective at recovering the target model's functionality, often reaching nearly 100% of the target's accuracy on downstream tasks. Furthermore, it exhibits robustness against common defenses such as embedding noise, output rounding, and coordinate suppression. Interestingly, the refinement process also helps the surrogate "unlearn" watermarks, making detection more difficult for defenders.
Conclusion
RDA presents a more accurate and cost-effective paradigm for stealing SSL encoders. By decoupling the query and training phases through sample-wise prototypes, it achieves state-of-the-art performance with minimal query overhead. The success of RDA highlights the inadequacy of current defense mechanisms and underscores the need for more advanced protection strategies for commercial SSL models.
Summary updated