Introduction
Controlling the attributes and safety of Large Language Model (LLM) generations is a critical challenge in AI alignment. Current approaches typically fall into two categories: simple prompting, which often fails to enforce negative constraints (e.g., "don't be toxic"), and auxiliary-model methods (like FUDGE), which require specialized training and significant computational overhead.
PREADD (Prefix-Adaptive Decoding) addresses this gap by providing a lightweight, training-free method for steering LLM outputs. By operating at the logit level, PREADD allows researchers to precisely tune the strength of attributes like toxicity, bias, or sentiment using a single hyperparameter, effectively generalizing standard prompting into a more powerful and adjustable framework.
Methodology
PREADD operates during the decoding process by contrasting the internal logit distributions of a model under two different conditions: the original "raw" prompt and a "prefix-prepended" prompt that contains a specific control instruction (e.g., "The following text is toxic...").
The core mechanism involves calculating the difference ($d$) between the log-probabilities of these two prompts. This difference represents the "direction" the prefix is pushing the model. By adding $\alpha d$ back into the original logits—where $\alpha$ is a user-defined strength parameter—the model can be steered toward or away from the attribute.
One of the method's key strengths is its ability to handle negative constraints. To reduce toxicity, PREADD uses a toxicity-encouraging prefix and sets $\alpha$ to a negative value. This mathematically subtracts the unwanted behavior from the distribution, telling the model to do the opposite of what a toxic model would do.
Experimental Setup
The researchers evaluated PREADD across three distinct tasks:
- Toxicity Mitigation: Using the RealToxicityPrompts dataset to steer away from toxic continuations.
- Gender Bias Reduction: Using WinoBias to test the reduction of gender stereotypes in occupational contexts.
- Sentiment Control: Using the Stanford IMDb dataset to flip review sentiment.
The study compared two variants: PREADD-S (using a static prefix) and PREADD-D (which dynamically selects the best contrastive prefix from a datastore based on context).
Key Results
PREADD demonstrated superior performance across all metrics, outperforming both standard prompting and auxiliary-expert methods without requiring any additional training.
In toxicity mitigation, PREADD-S and PREADD-D reduced toxicity by 12% or more relative to both standard prompting and the auxiliary-expert method (FUDGE) on the most difficult prompt sets. On the WinoBias task, PREADD achieved a 20% relative reduction in gender bias compared to the base model. Furthermore, in sentiment tasks, PREADD significantly increased the success rate of flipping sentiment (e.g., from 16.8% to 63.1% for positive-to-negative transitions).
Human validation confirmed these quantitative findings, noting that PREADD successfully reduced undesirable attributes without a significant drop in text fluency or relevance.
Discussion and Limitations
While PREADD offers a flexible and powerful control mechanism, it requires access to the model's internal logit distributions, which may limit its application with closed-source APIs. Additionally, setting the control strength ($\alpha$) too high (typically $> 2.5$) can lead to nonsensical or degenerate token generation. Future work may explore extending PREADD to satisfy multiple simultaneous constraints and testing its efficacy in non-English contexts.
Summary updated