Abstract
Large-scale pre-trained language models have displayed unrivaled capacity in generating text that closely resembles human-written text. Nevertheless, generating texts adhering to specific conditions without fine-tuning or adding new parameters can be challenging. Contemporary approaches commonly rely on either prompts or auxiliary models to avoid modifying the language models. These auxiliary models are designed to assess whether a generated token contributes to meeting the desired requirements. These approaches adjust the distribution of the next token during the inference phase by leveraging the prediction score of the desired attribute to calculate gradients. However, these auxiliary models typically require the language model’s latent states. This prerequisite challenges integrating various existing black box attribute models or tools. We present the Plug-in Language Model (PiLM) as a solution to address the limitations. PiLM leverages reinforcement learning to utilize black box tools directly, adjusting the latent state to control text generation. However, performing backpropagation during the inference phase is time-consuming for PiLM. By replacing backpropagation with a simple regression model, PiLM can achieve an inference time comparable to that of the original LLM. Experiment results show that our approaches in this paper outperform existing state-of-the-art methods that rely on gradient-based, weighted decoding, or prompt-based methodologies.
Introduction
While Large Language Models (LLMs) are incredibly fluent, steering them to adhere to specific constraints—such as maintaining a positive tone or avoiding toxic language—remains a significant challenge. Existing solutions typically fall into two categories: gradient-based methods and weighted decoding.
Gradient-based methods, such as PPLM, adjust the model's internal latent states using backpropagation during inference. However, this process is computationally expensive, requiring a "mini-training" step for every generated token. Weighted decoding methods, like FUDGE, adjust token probabilities based on external classifiers. While faster, these often ignore the model's internal logic, leading to repetitive or ungrammatical outputs. Furthermore, most current methods require "white-box" access to attribute classifiers, which prevents the use of high-quality, proprietary, or non-differentiable black-box tools.
Methodology
To address these limitations, the authors propose the Plug-in Language Model (PiLM), a two-stage framework that enables precise control without sacrificing speed or fluency.
Stage 1: PiLM-RL (The Teacher)
Instead of relying on gradients, PiLM-RL uses Reinforcement Learning (RL). During a warm-up phase, the model generates potential future sequences ($n$ tokens ahead). A black-box tool evaluates these sequences and provides a reward. PiLM then uses Policy Gradient to update the model's latent states (specifically the Key-Value pairs in the Transformer) to maximize that reward. This look-ahead mechanism allows the model to understand how current state adjustments affect future coherence.
Stage 2: PiLM-Controller (The Shortcut)
Because RL optimization is too slow for real-time applications, the authors introduce the Latent Controller. They treat the difference between the original latent state and the RL-optimized state as a regression problem. A simple 2-layer fully connected neural network is trained to predict the optimized state directly from the original one. At inference time, the LLM's internal state is passed through this tiny plug-in controller, eliminating the need for backpropagation or look-ahead generation.
Experimental Setup
The researchers evaluated PiLM across three primary tasks using standard benchmarks:
- Sentiment Control: Utilizing the Yelp polarity dataset and a BERT-based classifier for evaluation.
- Topic Control: Steering the model toward subjects like "Science" or "Politics" using established wordlists.
- Language Detoxification: Reducing toxic output using the RealToxicityPrompts dataset and the Perspective API.
- Fluency and Grammar: Evaluated using LLaMA-2-7B for perplexity and a RoBERTa model trained on the Corpus of Linguistic Acceptability (CoLA).
Results and Discussion
PiLM-Controller represents a significant advancement in efficient controlled generation. It achieves an inference speed approximately 27x faster than gradient-based PPLM, performing nearly as fast as an unconditional GPT-2 model.
Key Insights
- Regression as Optimization: The "shift" required to modify a latent state for a specific attribute is predictable. A simple MLP can learn this mapping, effectively caching the results of expensive optimization.
- Future-Context Coherence: By looking $n$ tokens into the future during the RL phase, PiLM avoids the "short-sighted" choices that often lead to ungrammatical outputs in other controllers.
- Black-Box Flexibility: The use of RL rewards rather than gradients allows PiLM to integrate with any external tool, including proprietary APIs or non-differentiable scripts.
In sentiment tasks, PiLM achieved a success rate of 0.99 for positive and 0.97 for negative sentiment, significantly outperforming PPLM (0.79/0.58) while maintaining superior fluency and lower perplexity.
Conclusion and Future Work
PiLM offers a robust framework for steering LLMs with minimal overhead. However, it requires access to the model's internal latent states, making it incompatible with API-only models like GPT-4. Future research will explore multi-attribute control and the use of this framework to mitigate hallucinations by grounding latent states in input documents.
Summary updated