Deconstructing Classifiers: Towards A Data Reconstruction Attack Against Text Classification Models

Adel Elmahdy, Ahmed SalemACL 2024

TL;DR

This paper introduces the "Mix And Match" attack, a novel method to reconstruct training data from text classification models that are fine-tuned from large language models (LLMs). Unlike previous attacks that focused on generative LLMs, this work demonstrates that even classification models are vulnerable, highlighting a significant privacy risk.

What Problem Does This Paper Address?

Natural Language Processing (NLP) models, particularly those used for text classification (like spam detection or sentiment analysis), are increasingly built by fine-tuning pre-trained Large Language Models (LLMs). While this approach is efficient, it raises privacy concerns. Specifically, data reconstruction attacks aim to extract the sensitive data used to train these models. Previous research primarily focused on the privacy risks of generative LLMs, assuming classification models were more secure. This paper addresses this gap by proposing and evaluating a new data reconstruction attack specifically targeting these fine-tuned classification models, demonstrating their vulnerability to data leakage.

Methods

The core of the proposed attack is the "Mix And Match" method, which operates in two phases:

  • Candidate Generation Phase: Instead of exhaustively searching the entire vocabulary for potential tokens, this phase leverages the generative capabilities of the base LLM (before fine-tuning). The classification head of the target model is replaced with the original generation head of the LLM, creating a "Frankenstein model." By masking a target token in a "canary" sentence (a specially crafted piece of text inserted into the training data), this Frankenstein model generates a smaller, more manageable set of candidate tokens.
  • Candidate Pruning Phase: The candidate tokens generated in the first phase are then filtered and ranked. Initial filtering removes incomplete tokens and punctuation. Subsequently, the original classification head is reattached, and a membership inference attack (specifically, a simple loss-based attack) is used to score each candidate token. The token with the lowest loss is considered the most likely to be the original, reconstructed token.

The authors also explore different types of "canaries" (organic vs. random, high vs. low frequency tokens, varying lengths) and their impact on the attack's success.

Datasets

The authors evaluate their attack on two datasets:

  • Yelp Reviews Dataset: Used for topic classification, assigning one of five star ratings to reviews.
  • Reddit Dataset: Used for topic classification, predicting the subreddit associated with a user comment.

For training, they sampled 10,000 data points, and for validation, 2,500 data points. No new benchmarks were introduced; they used existing datasets for evaluation.

Key Insights and Ideas

  • Classification Models are Vulnerable: The most significant insight is that text classification models, even when fine-tuned from LLMs, are susceptible to data reconstruction attacks, contrary to prior assumptions.
  • Efficiency through Generation: The "Mix And Match" attack significantly improves efficiency by using the LLM's generative power to create a smaller set of candidate tokens, avoiding the computational cost of exhaustive search.
  • Canary Design Matters: The type and characteristics of the "canary" sentences (organic vs. random, token frequency, length) critically influence the attack's success rate.
  • Frankenstein Model Advantage: The "Frankenstein model" (base LLM + generation head) generally outperforms using just the base LLM for candidate generation, especially with organic canaries.

Key Results

  • The "Mix And Match" attack, using the Frankenstein model, generates up to 50x fewer candidate tokens compared to an exhaustive search approach.
  • The attack successfully reconstructs target tokens, with performance varying based on canary type and repetition. For instance, higher repetition of canaries generally leads to better reconstruction.
  • The Frankenstein model generally outperforms the base language model in candidate pruning, especially for organic canaries.
  • The study shows that mislabeling canaries (adversarial manipulation) can significantly enhance the attack's effectiveness.

Limitations and Future Work

  • Target Token Constraint: The current methodology is primarily designed to reconstruct a single target token. The authors acknowledge that extending this to reconstruct multiple tokens is a challenge for future work.
  • LLM Fine-tuning Focus: The attack is specifically designed for classification models fine-tuned on top of LLMs. While this is a common practice, it limits the attack's applicability to other model types.
  • Future Directions: The authors suggest exploring the attack on non-masked LLMs (like GPT models), incorporating intermediate layers (adapters) for better connectivity, and leveraging prompt-based learning to optimize the generation process. They also highlight the need for robust defenses against membership inference attacks, as they are a key component of the pruning phase.

Summary updated

BibTeX
@inproceedings{elmahdy2024deconstructing,
  author = {Adel Elmahdy and Ahmed Salem},
  title = {{Deconstructing Classifiers: Towards A Data Reconstruction Attack Against Text Classification Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure