PREADD: Prefix-Adaptive Decoding for Controlled Text Generation

Jonathan Pei, Kevin Yang, Dan Klein

Introduction

Controlling the attributes and safety of Large Language Model (LLM) generations is a critical challenge in AI alignment. Current approaches typically fall into two categories: simple prompting, which often fails to enforce negative constraints (e.g., "don't be toxic"), and auxiliary-model methods (like FUDGE), which require specialized training and significant computational overhead.

PREADD (Prefix-Adaptive Decoding) addresses this gap by providing a lightweight, training-free method for steering LLM outputs. By operating at the logit level, PREADD allows researchers to precisely tune the strength of attributes like toxicity, bias, or sentiment using a single hyperparameter, effectively generalizing standard prompting into a more powerful and adjustable framework.

Methodology

PREADD operates during the decoding process by contrasting the internal logit distributions of a model under two different conditions: the original "raw" prompt and a "prefix-prepended" prompt that contains a specific control instruction (e.g., "The following text is toxic...").

The PREADD decoding process contrasts raw and prefixed prompts to steer model output.

The core mechanism involves calculating the difference ($d$) between the log-probabilities of these two prompts. This difference represents the "direction" the prefix is pushing the model. By adding $\alpha d$ back into the original logits—where $\alpha$ is a user-defined strength parameter—the model can be steered toward or away from the attribute.

One of the method's key strengths is its ability to handle negative constraints. To reduce toxicity, PREADD uses a toxicity-encouraging prefix and sets $\alpha$ to a negative value. This mathematically subtracts the unwanted behavior from the distribution, telling the model to do the opposite of what a toxic model would do.

Experimental Setup

The researchers evaluated PREADD across three distinct tasks:

  • Toxicity Mitigation: Using the RealToxicityPrompts dataset to steer away from toxic continuations.
  • Gender Bias Reduction: Using WinoBias to test the reduction of gender stereotypes in occupational contexts.
  • Sentiment Control: Using the Stanford IMDb dataset to flip review sentiment.

The study compared two variants: PREADD-S (using a static prefix) and PREADD-D (which dynamically selects the best contrastive prefix from a datastore based on context).

Key Results

PREADD demonstrated superior performance across all metrics, outperforming both standard prompting and auxiliary-expert methods without requiring any additional training.

Toxicity mitigation results comparing PREADD variants against standard prompting and auxiliary models.

In toxicity mitigation, PREADD-S and PREADD-D reduced toxicity by 12% or more relative to both standard prompting and the auxiliary-expert method (FUDGE) on the most difficult prompt sets. On the WinoBias task, PREADD achieved a 20% relative reduction in gender bias compared to the base model. Furthermore, in sentiment tasks, PREADD significantly increased the success rate of flipping sentiment (e.g., from 16.8% to 63.1% for positive-to-negative transitions).

Human validation confirmed these quantitative findings, noting that PREADD successfully reduced undesirable attributes without a significant drop in text fluency or relevance.

Discussion and Limitations

While PREADD offers a flexible and powerful control mechanism, it requires access to the model's internal logit distributions, which may limit its application with closed-source APIs. Additionally, setting the control strength ($\alpha$) too high (typically $> 2.5$) can lead to nonsensical or degenerate token generation. Future work may explore extending PREADD to satisfy multiple simultaneous constraints and testing its efficacy in non-English contexts.

Summary updated

Abstract

We propose Prefix-Adaptive Decoding (PREADD), a flexible method for controlled text generation. Unlike existing methods that use auxiliary expert models to control for attributes, PREADD does not require an external model, instead relying on linearly combining output logits from multiple prompts. Specifically, PREADD contrasts the output logits generated using a raw prompt against those generated using a prefix-prepended prompt, enabling both positive and negative control with respect to any attribute encapsulated by the prefix. We evaluate PREADD on three tasks -- toxic output mitigation, gender bias reduction, and sentiment control -- and find that PREADD outperforms not only prompting baselines, but also an auxiliary-expert control method, by 12% or more in relative gain on our main metrics for each task.
BibTeX
@inproceedings{pei2023preadd,
  author = {Jonathan Pei and Kevin Yang and Dan Klein},
  title = {{PREADD: Prefix-Adaptive Decoding for Controlled Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=ohvcvLH3FR7},
}
Figure