Abstract
Controlled text generation is essential for tailoring large language models (LLMs) to specific tasks, yet existing methods often suffer from high computational costs or degradation in text quality. We present FUDGE (Future Discriminators for Generation), a lightweight, modular framework that steers pre-trained models without requiring fine-tuning or gradient-based optimization. By employing a binary classifier to predict whether a partial sequence will eventually satisfy a desired attribute, FUDGE re-weights the model's output probabilities in real-time. Our results demonstrate that FUDGE significantly outperforms baselines in poetry generation, topic control, and formality transfer while maintaining high fluency and efficiency.
Introduction
While modern LLMs excel at generating fluent text, "steering" them toward specific attributes—such as a particular poetic meter, rhyme scheme, or level of formality—remains a significant challenge. Traditional approaches typically fall into three categories: fine-tuning, which is computationally expensive and risks "distribution shift"; rejection sampling, which is highly inefficient; and gradient-based methods like PPLM, which are slow and require access to internal model weights.
FUDGE addresses these limitations by providing a method that is modular, efficient, and flexible. It requires only the output probabilities of the base model, making it compatible with "API-only" proprietary models.
Methodology: Steering with Future Discriminators
The core innovation of FUDGE is the application of Bayes' Rule to look ahead during the generation process. Instead of evaluating the current token in isolation, FUDGE asks: "Given the words written so far, how likely is it that this sentence will eventually satisfy the target attribute?"
The Inference Process
FUDGE decomposes the conditional probability $P(x_i | x_{1:i-1}, a)$ into the product of the base model's probability $P(x_i | x_{1:i-1})$ and a "future discriminator" $P(a | x_{1:i})$. The discriminator is a lightweight binary classifier (typically an LSTM) trained on text prefixes to predict attribute satisfaction. At each step, FUDGE filters the top candidate words from the base model through this discriminator, multiplying and renormalizing the probabilities to steer the output toward the goal.
This approach allows for multi-constraint generation. Because FUDGE operates in log-probability space, multiple discriminators (e.g., for both topic and rhyme) can be combined simply by summing their log-probabilities.
Experimental Results
The authors evaluated FUDGE across three distinct domains: poetry, topic control, and machine translation.
Poetry and Topic Control
In poetry generation (couplet completion), FUDGE achieved a 44% success rate in generating valid iambic pentameter, nearly matching the performance of Shakespearean text (45%) and vastly outperforming the gradient-based PPLM baseline (0%).
In topic control tasks, FUDGE maintained higher fluency (4.30/5.0) than PPLM (4.05/5.0) while consistently generating on-topic content. Notably, FUDGE is highly efficient, generating 420 samples in just 15 minutes on a single GPU— a task that took PPLM several hours.
Formality in Machine Translation
For zero-shot formality transfer in Spanish-English translation, FUDGE successfully increased formality scores without the typical drop in translation quality (BLEU scores) seen in standard style-transfer baselines. This suggests that FUDGE preserves the underlying model's linguistic knowledge while effectively re-weighting its vocabulary.
Discussion and Limitations
FUDGE offers a powerful mechanism for zero-weight control, ensuring that the base model's fundamental understanding of grammar remains intact. However, the method is probabilistic and does not provide hard guarantees for constraints like exact syllable counts. Additionally, training the future discriminator requires a labeled dataset of the target attribute. Future work may explore combining FUDGE with reranking or rejection sampling to further tighten control for high-stakes applications.
Summary updated