Controllable Text Generation with Residual Memory Transformer

Hanqing Zhang, Si Sun, Haiming Wu, Dawei SongACL 2024 · DOI

Abstract

Controllable Text Generation (CTG) is essential for aligning Large Language Models (LLMs) with specific constraints such as keyword inclusion, sentiment, or length. However, existing approaches often face a "trilemma" between flexibility, control granularity, and computational efficiency. We introduce the Residual Memory Transformer (RMT), a non-intrusive, lightweight plugin that steers frozen LLMs using a residual learning paradigm. RMT achieves state-of-the-art results on keyword inclusion tasks, outperforming models like GPT-3.5-turbo, while maintaining the inference speed of the base model.

Introduction

The rapid advancement of Large Language Models (LLMs) has revolutionized natural language processing, yet precisely steering their output remains a challenge. Controllable Text Generation (CTG) requires models to adhere to fine-grained constraints—ranging from lexical requirements (e.g., "include specific keywords") to stylistic attributes (e.g., "maintain a positive sentiment").

Current CTG methods typically fall into three categories: prompt-based, fine-tuning, or post-processing. Prompt-based methods often lose efficacy over long sequences. Fine-tuning is computationally expensive and risks "catastrophic forgetting" of the model's original capabilities. Post-processing methods, such as PPLM, are notoriously slow due to iterative optimization at each decoding step. RMT addresses these issues by acting as a "best of all worlds" solution: a flexible, granular, and efficient plugin.

Methodology: The Residual Memory Transformer

The core of RMT is an encoder-decoder architecture designed to sit alongside a frozen Causal Language Model (CLM), such as GPT-2 or Llama-2.

Residual Learning Paradigm

Unlike traditional adapters that modify internal layers, RMT operates at the output level. It calculates a "control" probability distribution ($y^c_t$) based on the input constraints and adds it to the base model's distribution ($y^g_t$). This "late fusion" ensures that the base model's generative power is preserved while the plugin provides the necessary steering.

Architecture of the Residual Memory Transformer (RMT) showing the late fusion of control signals with the base CLM.

Architectural Components

The RMT decoder utilizes three distinct attention mechanisms:

  1. Causal Self-Attention: Standard attention over previously generated tokens in the plugin.
  2. Causal CLM Attention: A "memory" mechanism that allows the plugin to attend to the hidden states of the frozen base model, ensuring contextual coherence.
  3. Cross Control Attention: Ensures the control signal (from the encoder) is consistently applied throughout the generation process, preventing the "fading" effect common in prompt-based steering.

Training Strategy

RMT is trained in two stages. First, a Denoising Pre-training phase on 4 million Wikipedia sentences teaches the model the relationship between keywords and coherent text. Second, the model is Fine-tuned for specific tasks like sentiment or length control using standard Maximum Likelihood Estimation (MLE).

Experimental Evaluation

The authors evaluated RMT across several benchmarks, including CommonGen (generative commonsense reasoning), C2Gen (context-aware keyword inclusion), and SST-5 (sentiment control).

Key Results

RMT demonstrates significant improvements over existing baselines. On the C2Gen task, which requires generating text that includes specific keywords within a given context, RMT achieved a coverage score of 91.8%. This significantly outperforms GPT-3.5-turbo (82.2%) and specialized models like NRP (81.0%).

Performance comparison on CommonGen and C2Gen benchmarks, highlighting RMT's superior keyword coverage.

Beyond keyword inclusion, RMT excels in:

  • Sentiment Steering: Achieving 97.6% correctness in positive sentiment generation on SST-5.
  • Length Control: Successfully generating sentences within ±1 word of a target length, a task where standard LLMs often struggle.
  • Efficiency: RMT adds minimal latency. For GPT-2, inference time only increased from 0.78s to 0.88s, whereas post-processing methods can take orders of magnitude longer.

Discussion and Insights

RMT's success stems from its non-intrusive nature. By keeping the base LLM frozen, it avoids the degradation of general language abilities. Furthermore, the model is highly efficient; a 3-layer RMT (tuning roughly 16% of the base model's parameters) is sufficient for effective control. The use of cross-attention effectively eliminates "distance bias," allowing for consistent control regardless of sequence length.

Limitations and Future Work

Despite its performance, RMT requires access to the base model's internal hidden states and output logits, limiting its use with closed-source APIs like GPT-4. Additionally, while it excels at constraint satisfaction, it can occasionally produce grammatically correct but logically strained sentences to meet difficult keyword requirements. Future research may explore integrating RMT with Retrieval-Augmented Generation (RAG) or extending its capabilities to multi-modal control.

Summary updated

BibTeX
@inproceedings{zhang2024controllable,
  author = {Hanqing Zhang and Si Sun and Haiming Wu and Dawei Song},
  title = {{Controllable Text Generation with Residual Memory Transformer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.62},
}
Figure