disco: a toolkit for Distributional Control of Generative Models

Germán Kruszewski, Jos Rozen, Marc Dymetman

Introduction

While pre-trained generative models, such as Large Language Models (LLMs) and diffusion models, have demonstrated remarkable capabilities, they often inherit biases from their training data or fail to adhere to specific stylistic and safety constraints. Traditional methods for steering these models—such as prompt engineering or Reinforcement Learning from Human Feedback (RLHF)—frequently prove brittle or complex to implement.

Specifically, enforcing distributional constraints (e.g., ensuring a specific percentage of outputs contain a certain feature) remains a significant challenge. The disco library addresses this by providing an open-source Python toolkit designed for Generation under Distributional Control (GDC). By decoupling the definition of a target distribution from the optimization process, disco allows developers to precisely control model outputs without the risk of catastrophic forgetting associated with standard fine-tuning.

Methodology

The GDC Framework

The disco toolkit is built upon the GDC framework, which conceptualizes distributional control as a two-step process:

  1. Defining the Target (The "What"): Users define the desired output characteristics using "Scorers" (classifiers or reward functions) and set specific "moments" or expectations. For instance, a user might specify that a binary feature should appear with a mean of 0.5 (present in half the samples). This mathematical definition forms a target Energy-Based Model (EBM).
  2. Approximating the Target (The "How"): To align a model with the EBM, disco utilizes the Distributional Policy Gradient (DPG) algorithm. This fine-tunes auto-regressive models to match the target distribution while maintaining proximity to the original base model to preserve general capabilities.

Toolkit Architecture

disco is integrated with the PyTorch and HuggingFace Transformers ecosystems, offering a unified API consisting of three primary components:

  • Samplers: Responsible for generating outputs from the model.
  • Scorers: Evaluate generated samples against defined constraints.
  • Tuners: Handle the optimization and fine-tuning process.

Additionally, the library introduces Quasi-Rejection Sampling (QRS). This inference-time method allows users to trade off sampling speed for higher fidelity to the target distribution, enabling near-perfect constraint satisfaction when necessary.

Experimental Evaluation

The authors demonstrated the versatility of disco through three distinct showcase experiments using models like GPT-2, T5, and BlenderBot.

1. Controlled Word Frequency (The "Amazing" Task)

In a controlled experiment to force GPT-2 to include the word "amazing," the DPG algorithm with a large batch size (4096) achieved a success rate of 57%. By applying QRS (beta=0.02) at inference time, the system achieved 100% success in including the target word while retaining 10% of the generated samples.

2. Reducing Hallucinations in Summarization

Using a T5 model on the CNN/DailyMail dataset, the toolkit was used to enforce factual consistency. By constraining the model to only include entities present in the source text, disco improved entity precision from 0.91 to 0.94 and significantly boosted recall from 0.26 to 0.45, effectively reducing "hallucinations" in the generated summaries.

3. Personality Shift in Chatbots

disco was applied to BlenderBot to shift its personality toward the "ESFP" (Extraverted, Sensing, Feeling, Perceiving) type. The toolkit successfully moved the model's "Extraversion" score from 0.59 to 0.64, demonstrating the ability to perform nuanced stylistic adjustments.

Discussion and Future Work

A key insight from the development of disco is that RLHF can be viewed as a specific instance of distributional control. disco generalizes this concept, allowing for multi-feature constraints that go beyond simple reward maximization. However, the effectiveness of the toolkit is inherently tied to the quality of the "Scorer"; if the underlying classifier is biased or inaccurate, the resulting model will reflect those flaws.

Future iterations of the toolkit aim to incorporate $f$-DPG, a generalized algorithm that supports any $f$-divergence. This is expected to offer improved efficiency and greater diversity in generated outputs, further lowering the barrier to precise generative control.

Summary updated

BibTeX
@inproceedings{kruszewski2023disco,
  author = {Germán Kruszewski and Jos Rozen and Marc Dymetman},
  title = {{disco: a toolkit for Distributional Control of Generative Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-demo.14},
}
Figure