A Distributional Approach to Controlled Text Generation

Muhammad Khalifa, Hady Elsahar, Marc Dymetman

Abstract

We propose a Distributional Approach for addressing Controlled Text Generation from pre-trained Language Models (LMs). This approach permits to specify, in a single formal framework, both "pointwise" and "distributional" constraints over the target LM -- to our knowledge, the first model with such generality -- while minimizing KL divergence from the initial LM distribution. The optimal target distribution is then uniquely determined as an explicit EBM (Energy-Based Model) representation. From that optimal representation we then train a target controlled Autoregressive LM through an adaptive distributional variant of Policy Gradient. We conduct a first set of experiments over pointwise constraints showing the advantages of our approach over a set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the initial LM. We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias in Language Models. Through an ablation study, we show the effectiveness of our adaptive technique for obtaining faster convergence.

Introduction

Pre-trained language models (LMs) like GPT-2 are powerful generators but often act as "stochastic parrots," mirroring biases and toxicity present in their training data. Controlling these models is essential for safe and effective deployment. Existing methods typically adopt a pointwise perspective, using Reinforcement Learning (RL) to reward individual sentences that satisfy specific criteria. However, this approach suffers from two major drawbacks:

  1. Degeneration: Models often exploit "loopholes" to maximize rewards, leading to repetitive and nonsensical text that lacks the fluency of the original LM.
  2. Collective Goal Inefficiency: Standard RL cannot easily enforce distributional goals, such as ensuring a balanced gender representation across a set of generated biographies, because rewards are calculated per-sentence rather than across the entire output distribution.

Methods: Generation with Distributional Control (GDC)

The proposed GDC framework addresses these issues by framing control as a divergence minimization problem. The process consists of two primary steps:

1. From Constraints to Energy-Based Models (EBM)

Desiderata are formalized as "moment constraints" (mathematical expectations). Using principles from information geometry, the authors demonstrate that the unique distribution satisfying these constraints while remaining closest to the original LM is an Energy-Based Model (EBM). In this view, the original LM serves as a prior, and the constraints act as forces reshaping the distribution. This relationship is governed by a "Pythagorean Identity," providing a firm theoretical foundation for the uniqueness and optimality of the solution.

Geometric intuition of the Pythagorean identity in the GDC framework.

2. From EBM to Autoregressive Policy

Since sampling directly from an EBM is computationally difficult, GDC trains a standard autoregressive model (the "policy") to approximate this ideal distribution. To overcome the failure of standard importance sampling when the current model diverges significantly from the target, the authors introduce KL-adaptive Distributional Policy Gradient (DPG). This algorithm periodically updates its proposal distribution to match its evolving progress, significantly accelerating convergence for rare constraints.

Experimental Setup

The researchers evaluated GDC using GPT-2 (small and medium) against several benchmarks:

  • Pointwise Tasks: Forcing specific words (e.g., "vampire"), topic-based word lists (e.g., "kitchen"), and sentiment/clickbait classifiers.
  • Distributional Tasks: A dataset of 700,000 Wikipedia biographies was used to demonstrate GDC's ability to mitigate demographic bias by balancing gender and profession distributions.

Key Results

GDC demonstrates significant improvements over baselines like REINFORCE and PPO-based methods (Ziegler et al.):

  • Superior Diversity: Unlike standard RL methods that often collapse into repeating a single high-reward sentence, GDC maintains a diverse vocabulary and sentence structure.
  • Effective De-biasing: In biography generation, where base GPT-2 produced only ~7% female biographies, GDC successfully shifted this to 36.7% (against a 50% target) while preserving fluency and style.
  • Convergence and Fluency: The adaptive technique proved vital for rare constraints. Furthermore, GDC achieved lower perplexity than PPLM and CTRL, indicating more natural text generation.
Comparison of diversity metrics (Self-BLEU-5 and Dist-1) across different methods.

Conclusion and Future Work

GDC provides a robust framework for controlled generation by decoupling the definition of the ideal distribution from the training of the policy. While highly effective, a "residual gap" remains in hitting 100% satisfaction for hard pointwise constraints within standard training limits. Future work may involve combining GDC with MCMC sampling at inference time or scaling the approach to larger models like GPT-3/4, which remains a significant engineering challenge due to computational intensity.

Code available at: https://github.com/naver/gdc

Summary updated

Abstract

We propose a Distributional Approach for addressing Controlled Text Generation from pre-trained Language Models (LMs). This approach permits to specify, in a single formal framework, both "pointwise" and "distributional" constraints over the target LM -- to our knowledge, the first model with such generality -- while minimizing KL divergence from the initial LM distribution. The optimal target distribution is then uniquely determined as an explicit EBM (Energy-Based Model) representation. From that optimal representation we then train a target controlled Autoregressive LM through an adaptive distributional variant of Policy Gradient. We conduct a first set of experiments over pointwise constraints showing the advantages of our approach over a set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the initial LM. We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias in Language Models. Through an ablation study, we show the effectiveness of our adaptive technique for obtaining faster convergence. (Code available at https://github.com/naver/gdc)
BibTeX
@inproceedings{khalifa2021distributional,
  author = {Muhammad Khalifa and Hady Elsahar and Marc Dymetman},
  title = {{A Distributional Approach to Controlled Text Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=YRN2dSJ9XX6},
}
Figure