CTRL: A Conditional Transformer Language Model for Controllable Generation

Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, Richard SocherarXiv 2019 · arXiv

Introduction

Large-scale language models have demonstrated remarkable fluency, yet they often lack a mechanism for precise control. A user might provide a prompt, but the model's output style—whether it resembles a technical manual, a news report, or a casual social media post—is largely left to chance. CTRL (Conditional Transformer Language model) addresses this by introducing "control codes" to steer generation.

CTRL is a 1.63 billion-parameter model trained on a massive 140GB dataset. By prepending natural metadata (like URLs or domain names) as control codes during training, the model learns to associate specific styles and tasks with these identifiers. This allows for explicit "steering" of the generated text without the need for task-specific fine-tuning that might otherwise degrade general performance.

Architecture and Methods

CTRL utilizes a standard Transformer-based decoder-only architecture. With 48 layers and 16 attention heads, it was designed to capture complex linguistic patterns across a vast array of domains.

Transformer Blocks

The model's internal structure relies on residual connections and layer normalization to maintain stable gradients during the training of its 1.63 billion parameters. The architecture follows the standard pre-layer normalization pattern common in modern Transformers.

The CTRL architecture utilizes standard Transformer blocks with LayerNorm and residual connections.

Control Codes

The defining feature of CTRL is the use of control codes. During the unsupervised training phase, every text sequence is prefixed with a code identifying its source (e.g., Wikipedia, Legal, Reviews, or r/science). This simple modification transforms the standard language modeling objective $p(x)$ into a conditional one: $p(x|c)$, where $c$ is the control code.

Penalized Sampling

To improve the quality of generated text, the authors introduced penalized sampling. Standard greedy sampling often results in repetitive loops, while stochastic methods like Top-k sampling can occasionally select low-probability tokens that lead to factual inaccuracies. Penalized sampling addresses this by discounting the scores of tokens that have already appeared in the sequence, forcing the model to progress through the narrative without getting stuck.

Training Data

The model was trained on a diverse 140 GB corpus, including:

  • Knowledge & Literature: Wikipedia (multiple languages) and Project Gutenberg.
  • Social Media: Data from 45 distinct subreddits.
  • Web & News: OpenWebText and various news archives.
  • Specialized Tasks: Amazon Reviews for sentiment, WMT for translation, and SQuAD/ELI5 for question-answering.

Key Insights

Metadata as Supervision

One of the paper's primary contributions is the realization that metadata—such as domain names or URLs—serves as a "free" supervisory signal. This naturally occurring structure provides sufficient context for the model to learn distinct styles and intents without manual labeling.

The Sampling Trade-off

Controllability is particularly vital for factual accuracy. As shown in the probability distributions below, even when a model assigns the highest probability to the correct answer, significant probability mass is often distributed among other plausible-sounding but incorrect tokens. Without the ability to steer the model, stochastic sampling risks selecting these "near-misses."

Next-token probability distributions for factual prompts. Even when the correct answer has the highest probability, other options still hold significant mass, making unguided sampling risky.

Zero-Shot Code Mixing

Interestingly, CTRL can combine control codes in ways not seen during training. For instance, combining a Diet code with a German translation code enables the model to generate nutrition-related content in German, demonstrating a form of zero-shot transfer.

Source Attribution

By applying Bayes' rule, the model's conditional probability $p(x|c)$ can be used to perform source attribution. Given a sentence $x$, the model can estimate $p(c|x)$, identifying which part of the training data the text most likely originated from.

Results and Limitations

CTRL demonstrates high-fidelity domain switching. The same prompt, such as "The weather is...", can result in a scientific report under r/science or a casual observation under a different social media code. It also handles fine-grained control, such as generating reviews based on a specific star rating.

However, the model is not without limitations. It inevitably learns the biases and cultural associations present in its training data. Furthermore, while domain-level control is robust, achieving entity-level or highly granular temporal control remains a challenge for future research. The authors conclude with a strong emphasis on the ethical implications of large-scale generation, advocating for responsible innovation and a clear code of conduct for model deployment.

Summary updated

Abstract

Large-scale language models show promising text generation capabilities, but users cannot easily control particular aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data via model-based source attribution. We have released multiple full-sized, pretrained versions of CTRL at https://github.com/salesforce/ctrl.
BibTeX
@misc{keskar2019ctrl,
  author = {Nitish Shirish Keskar and Bryan McCann and Lav R. Varshney and Caiming Xiong and Richard Socher},
  title = {{CTRL: A Conditional Transformer Language Model for Controllable Generation}},
  year = {2019},
  eprint = {1909.05858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.05858},
}
Figure