Threats to Pre-trained Language Models: Survey and Taxonomy

Shangwei Guo, Chunlong Xie, Jiwei Li, Lingjuan Lyu, Tianwei ZhangarXiv 2022 · arXiv

Abstract

Pre-trained Language Models (PTLMs) such as BERT and GPT have revolutionized Natural Language Processing through the "pre-train then fine-tune" paradigm. However, this efficiency introduces a complex, multi-stage attack surface. This paper provides the first comprehensive taxonomy of security and privacy threats targeting PTLMs, systematizing risks across the model lifecycle—from pre-training to inference. A key contribution is the identification of "transferability" properties that allow attacks to persist even after fine-tuning on clean data, posing significant challenges for model integrity and data privacy.

Introduction

The shift toward using massive foundation models has fundamentally changed how NLP applications are built. Instead of training from scratch, developers adapt PTLMs for specific downstream tasks. While this approach is efficient, it creates a centralized point of failure. If a base model is compromised, every application built upon it inherits its vulnerabilities. Despite the discovery of various individual attacks, the field has lacked a unified framework to categorize these threats. This survey addresses this gap by analyzing how the inherent properties of PTLMs, such as their generalization capabilities, can be exploited by malicious actors.

A Three-Dimensional Taxonomy

The authors propose a structured framework to characterize PTLM threats based on three dimensions: the pipeline phase, adversarial goals, and model transferability.

The PTLM Pipeline

The lifecycle is divided into three distinct phases:

  1. Pre-training: The phase where a Model Publisher creates the base model using massive unlabeled datasets.
  2. Fine-tuning: The phase where a Downstream Service Provider (DSP) adapts the PTLM for specific tasks like sentiment analysis or question answering using labeled data.
  3. Inferring: The final phase where end-users query the deployed model for predictions.

Adversarial Goals

Threats are broadly categorized into two types:

  • Integrity Threats: Attacks aimed at manipulating model outputs, such as backdoors (triggers that force specific incorrect outputs) or evasion (adversarial examples designed to fool the model at inference).
  • Privacy Threats: Attacks aimed at extracting sensitive information, including training data (data privacy) or the model's internal parameters and logic (model privacy).
Overview of the PTLM pipeline, attack goals, and transferability types.

Model Transferability

A novel concept introduced in this work is the dual nature of attack transferability:

  • Landscape Transferability: An attack successful on one downstream task can often be adapted to others because they share the same PTLM backbone.
  • Portrait Transferability: Malicious backdoors injected during pre-training are remarkably persistent; they survive the fine-tuning process even when the developer uses pristine data for the final task.

Surveying the Attack Surface

By synthesizing findings from dozens of studies across standard NLP benchmarks (e.g., GLUE, SQuAD, PAWS), the survey highlights several critical vulnerabilities.

The Malicious Publisher and Persistent Backdoors

One of the most severe threats is the "poisoned" foundation model. Because developers often download models from public repositories like Hugging Face, a malicious publisher can release a version of a model that contains a hidden trigger. Research shows that attacks like RIPPLe can achieve nearly 100% success rates in triggering target labels while maintaining standard accuracy on clean data. Due to portrait transferability, standard fine-tuning is insufficient to "wash away" these deep-seated malicious behaviors.

Privacy Leakage and Model Extraction

PTLMs are often "honest-but-curious," meaning they inadvertently memorize sensitive parts of their training data. Attackers can employ "Embedding Inversion" to reconstruct original sentences from numerical vectors. Furthermore, because PTLMs are valuable intellectual property, they are targets for "imitation attacks," where adversaries query a model's API to train a "shadow" model that mimics the original's behavior.

Summary of existing literature on PTLM threats categorized by attack type and phase.

Future Directions and Challenges

The survey identifies several open challenges for the research community:

  • Robustness-Utility Trade-off: Current privacy defenses, such as adding noise to embeddings, often degrade model performance. Finding a balance remains difficult.
  • Evolving Backdoor Tactics: As defenses like ONION improve at detecting simple triggers, attackers are moving toward context-aware triggers that are harder to filter.
  • Machine Unlearning: There is a growing need for fine-tuning methods that can effectively "forget" malicious or sensitive data from the pre-training phase without losing general linguistic knowledge.

In conclusion, as PTLMs become more integrated into critical infrastructure, understanding and mitigating these multi-stage threats is essential for building trustworthy AI systems.

Summary updated

Abstract

Pre-trained language models (PTLMs) have achieved great success and remarkable performance over a wide range of natural language processing (NLP) tasks. However, there are also growing concerns regarding the potential security issues in the adoption of PTLMs. In this survey, we comprehensively systematize recently discovered threats to PTLM systems and applications. We perform our attack characterization from three interesting perspectives. (1) We show threats can occur at different stages of the PTLM pipeline raised by different malicious entities. (2) We identify two types of model transferability (landscape, portrait) that facilitate attacks. (3) Based on the attack goals, we summarize four categories of attacks (backdoor, evasion, data privacy and model privacy). We also discuss some open problems and research directions. We believe our survey and taxonomy will inspire future studies towards secure and privacy-preserving PTLMs.
BibTeX
@misc{guo2022threats,
  author = {Shangwei Guo and Chunlong Xie and Jiwei Li and Lingjuan Lyu and Tianwei Zhang},
  title = {{Threats to Pre-trained Language Models: Survey and Taxonomy}},
  year = {2022},
  eprint = {2202.06862},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.06862},
}
Figure