A Distributional Approach to Controlled Text Generation

Muhammad Khalifa, Hady Elsahar, Marc Dymetman

Large-scale language models (LMs) like GPT-2 and GPT-3 are incredibly fluent but notoriously difficult to control. The authors of this paper introduce Generation with Distributional Control (GDC), a novel framework that allows researchers to steer language models using both "pointwise" constraints (e.g., "this specific sentence must be positive") and "distributional" constraints (e.g., "50% of generated biographies should be about women"). By framing control as a KL-divergence minimization problem, GDC achieves high-quality, diverse text while successfully mitigating social biases in pre-trained models.

The Challenge of Controlling Language Models

Standard Reinforcement Learning (RL) approaches to controlling LMs often suffer from "degeneration." The model finds loopholes to maximize rewards, resulting in repetitive, gibberish text that lacks the linguistic richness of the original pre-trained model.

Furthermore, most existing control methods are "pointwise," focusing solely on the properties of individual outputs. This makes them ill-equipped to handle collective statistical properties, such as ensuring a model doesn't exhibit demographic bias across a large set of generations. The core challenge is satisfying these aggregate constraints without drifting so far from the original LM that the text becomes unreadable.

The GDC Framework

The authors propose a two-step approach to solve this, decoupling the definition of the optimal distribution from the process of learning to sample from it.

1. From Constraints to EBM: They formalize controlled generation as finding a target distribution $p$ that satisfies specific "moment constraints" (expectations of features) while staying as close as possible to the original LM $a$. This is achieved by minimizing the Kullback-Leibler (KL) divergence, $D_{KL}(p || a)$. Mathematically, this projection leads to a unique solution in the form of an Energy-Based Model (EBM).

KL Divergence Projection

2. From EBM to Policy: Because sampling directly from an EBM is computationally difficult, the authors train an autoregressive policy $\pi_\theta$ to approximate it. They introduce a novel KL-adaptive Distributional Policy Gradient (DPG) algorithm. The "adaptive" mechanism is crucial: the algorithm periodically updates its proposal distribution to match its current best approximation. This significantly accelerates convergence, especially when trying to enforce rare constraints, by shifting the source toward the target during training.

Key Insights

  • A Unified Framework: The paper elegantly demonstrates that pointwise control is simply a special case of distributional control. For example, requiring a word to appear in every sentence is equivalent to setting the expected frequency of that word to 1.0.
  • The KL-Divergence "Yardstick": By explicitly minimizing the distance to the original LM, the model inherits the favorable linguistic qualities of the pre-trained data. This prevents the collapse into repetitive nonsense often seen in vanilla RL fine-tuning.
  • Remedying Bias: Distributional constraints are a natural fit for de-biasing. Instead of forcing every single sentence to be neutral, researchers can specify that the aggregate output should match a desired demographic distribution.

Experimental Results

The researchers evaluated GDC using GPT-2 as the base model, testing on standard benchmarks (word lists, sentiment control) and a dataset of 700k Wikipedia Biographies to test de-biasing.

  • Superior Diversity and Fluency: In pointwise experiments, GDC maintained significantly higher vocabulary richness and lower repetition compared to baselines like REINFORCE or PPO-based fine-tuning. It achieved much lower perplexity than baselines like PPLM and CTRL, producing more natural-sounding text without needing heavy-handed repetition penalties.
Performance Metrics Comparison
  • Effective De-biasing: In the biography task, the base model was heavily biased, generating only ~7% female biographies. GDC successfully shifted the female biography percentage to 36.7% while simultaneously adjusting profession distributions (e.g., increasing "Science" and "Art" mentions) using a single training procedure.

Limitations and Future Work

While GDC represents a significant step forward, it doesn't always reach 100% constraint satisfaction for extremely hard pointwise requirements (often hovering around 80-90% for very rare words). The authors suggest that future work could bridge this "residual gap" by combining their policy with Markov Chain Monte Carlo (MCMC) sampling at inference time. Additionally, the computational cost of fine-tuning the policy for each new set of constraints is higher than "plug-and-play" methods that operate without retraining, presenting an area for future optimization.

Summary updated

Abstract

We propose a Distributional Approach for addressing Controlled Text Generation from pre-trained Language Models (LM). This approach permits to specify, in a single formal framework, both “pointwise’” and “distributional” constraints over the target LM — to our knowledge, the first model with such generality —while minimizing KL divergence from the initial LM distribution. The optimal target distribution is then uniquely determined as an explicit EBM (Energy-BasedModel) representation. From that optimal representation, we then train a target controlled Autoregressive LM through an adaptive distributional variant of PolicyGradient. We conduct a first set of experiments over pointwise constraints showing the advantages of our approach over a set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the pretrained LM. We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias in Language Models. Through an ablation study, we show the effectiveness of our adaptive technique for obtaining faster convergence. Code available at https://github.com/naver/gdc
BibTeX
@inproceedings{khalifa2021distributional,
  author = {Muhammad Khalifa and Hady Elsahar and Marc Dymetman},
  title = {{A Distributional Approach to Controlled Text Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=YRN2dSJ9XX6},
}
Figure