Towards Reverse Engineering of Language Models: A Survey

Xinpeng Ti, Wentao Ye, Zhifang Zhang, Junbo Zhao, Chang Yao, Lei Feng, Haobo WangEMNLP 2025 · OpenReview

TL;DR

This paper provides a comprehensive survey of "reverse engineering" (RE) techniques applied to large language models (LLMs), categorizing attacks into data recovery, model reconstruction, and defense strategies. It highlights the growing security concerns surrounding LLMs due to their increasing adoption and the proprietary nature of their development, emphasizing that even black-box models are vulnerable.

What Problem Does This Paper Address?

The rapid advancement and widespread deployment of large language models (LLMs) have made them invaluable tools across numerous fields. However, the immense computational resources and proprietary data required for their development make them attractive targets for theft and unauthorized access. Protecting the intellectual property embedded within these models and their training data is a critical concern. This paper addresses the need for a systematic understanding of the various methods used to reverse engineer LLMs, aiming to inform both the development of more robust defenses and the broader research community about these evolving security threats.

Methods

The paper introduces the concept of "Language Model Reverse Engineering" (LM RE) as the process of inferring key construction elements of LMs from their externally observable information. They categorize RE techniques into three main engines:

  • Data Recovery Engine: Focuses on extracting information about the training dataset. This includes:
  • Direct Recovery: Recovering specific training samples or run-time inputs.
  • Indirect Recovery: Inferring higher-level traits like membership inference functions or statistical properties.
  • Model Reconstruction Engine: Aims to rebuild the model's parameters, architecture, and functions. This is further broken down into:
  • Parameter Extraction: Stealing model weights, biases, hyperparameters, and algorithmic choices.
  • Function Imitation: Training an imitation model that replicates the victim model's knowledge and functionality, often using knowledge distillation or query-based approaches.
  • Structure Trace: Inferring structural information like hierarchical structure, scale, and architecture.
  • Defense Engine: Encompasses strategies to protect LLMs. These are divided into:
  • Preventive Defenses: Directly hardening the model against attacks (e.g., differential privacy, adversarial training, query throttling, targeted defenses).
  • Detective Defenses: Enhancing traceability and forensic capabilities to detect and attribute misuse (e.g., model fingerprinting).

The paper also formalizes the threat model based on different levels of adversary access (black-box, grey-box, white-box) and discusses various evaluation metrics and datasets used in RE research.

Datasets

The paper references several datasets commonly used in reverse engineering research for LLMs, including:

  • WikiMIA: Used for membership inference attacks, with sentence-level splitting based on date.
  • StackMIAsub: Similar to WikiMIA, with sentence-level splitting.
  • MIMIR: Used for membership inference attacks, with sentence-level splitting based on original training/test sets.
  • ArXiv - Document: Used for membership inference attacks, with document-level splitting.
  • LM Extraction Benchmark: A dataset for language model extraction.
  • Alpaca Code Generation: Provides code prompts for extraction.
  • Awesome-ChatGPT-Prompts: A collection of prompts designed to elicit specific behaviors from LLMs.
  • Unnatural Instructions: A diverse set of instructions for model interaction.

The authors also mention their own experiments using the "awesome-chatgpt-prompts" dataset and the Pythia-1.4B model for evaluating prompt extraction and membership inference attacks.

Key Insights and Ideas

  • Ubiquitous Vulnerability: Even "black-box" LLMs, accessible only through APIs, are susceptible to various reverse engineering attacks, posing significant risks to model owners.
  • Evolving Attack Landscape: The rapid evolution of LLMs necessitates continuous research into new attack and defense mechanisms, as older techniques may become obsolete.
  • Data vs. Model RE: The paper clearly distinguishes between attacks targeting the training data and those aiming to reconstruct the model itself, highlighting different methodologies and challenges for each.
  • Defense is a Moving Target: While preventive and detective defenses exist, attackers are constantly developing new methods, and even robust defenses can be bypassed. The paper notes that OpenAI is actively deploying countermeasures against effective prompt extraction techniques.
  • MIA Challenges: Membership inference attacks, while a significant area of research, face challenges with large-scale models due to distributional drift and excessive pretraining scale, often yielding near-random performance on general benchmarks.
  • Multimodal LLMs as New Frontiers: The emergence of multimodal LLMs introduces new attack surfaces and challenges for reverse engineering, requiring specialized research.

Key Results

The paper presents a survey and analysis rather than novel experimental results. However, it highlights findings from existing research:

  • Prompt Extraction Success: Experiments mentioned in the paper indicate that prompt extraction techniques achieve substantially higher success rates compared to data recovery methods.
  • Resilience of Reasoning Models: Reasoning-oriented LLMs exhibit greater resilience to prompt extraction attacks, with prompts that succeed against models like GPT-4o being ineffective against them.
  • MIA Limitations: The paper points out that many MIA techniques perform poorly on large-scale models and often rely on dataset-specific biases or improper membership splitting, leading to unreliable results.
  • Experimental Findings: The authors' own experiments showed that data extraction techniques failed to recover meaningful information from state-of-the-art models, yielding virtually no outputs resembling original training data.

Limitations and Future Work

Limitations:

  • The authors acknowledge that their survey, while extensive, may have overlooked some valuable contributions due to the vastness of the field.
  • Model reverse engineering is a broad topic encompassing various modalities (images, audio, text), and this survey primarily focuses on language models.

Future Work:

  • Enhancing Performance on Latest Models: Developing RE methods that can keep pace with the rapid evolution of LLM architectures and parameters.
  • Practical Challenges in Real-World Settings: Addressing constraints like controlled cost budgets, balancing attack effectiveness with complexity, and designing customized defenses for specific attack types.
  • Comprehensive Evaluation Framework: Establishing standardized benchmarks and protocols for evaluating RE methods, especially for data recovery, to enable more reliable comparisons.
  • Reverse Engineering in Multimodal Models: Investigating data recovery and model reconstruction in cross-modal settings and designing robust protective mechanisms for multimodal LLMs.
  • Benchmarking and Open-Source Tools: The paper mentions an appendix cataloging open-source RE techniques and their performance on commercial systems, suggesting a need for continued efforts in this area.

Summary updated

Abstract

With the continuous development of language models and the widespread availability of various types of accessible interfaces, large language models (LLMs) have been applied to an increasing number of fields. However, due to the vast amounts of data and computational resources required for model development, protecting the model’s parameters and training data has become an urgent and crucial concern. Due to the revolutionary training and application paradigms of LLMs, many new attacks on language models have emerged in recent years. In this paper, we define these attacks as “reverse engineering” (RE) techniques on LMs and aim to provide an in-depth analysis of reverse engineering of language models. We illustrate various methods of reverse engineering applied to different aspects of a model, while also providing an introduction to existing protective strategies. On the one hand, it demonstrates the vulnerabilities of even black box models to different types of attacks; on the other hand, it offers a more holistic perspective for the development of new protective strategies for models.
BibTeX
@inproceedings{ti2025towards,
  author = {Xinpeng Ti and Wentao Ye and Zhifang Zhang and Junbo Zhao and Chang Yao and Lei Feng and Haobo Wang},
  title = {{Towards Reverse Engineering of Language Models: A Survey}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=9D1WzVYIJt},
}
Figure