Abstract
This work introduces Focused-Variation Network (FVN), a novel model to control language generation. The main problems in previous controlled language generation models range from the difficulty of generating text according to the given attributes, to the lack of diversity of the generated texts. FVN addresses these issues by learning disjoint discrete latent spaces for each attribute inside codebooks, which allows for both controllability and diversity, while at the same time generating fluent text. We evaluate FVN on two text generation datasets with annotated content and style, and show state-of-the-art performance as assessed by automatic and human evaluations.
Introduction
While modern language models are excellent at generating fluent text, "steering" them remains a significant challenge. Researchers often struggle with two conflicting goals: controllability (ensuring the model includes specific data points and adopts a specific persona) and diversity (ensuring the model doesn't just output repetitive, "safe" sentences).
Previous approaches like Conditional Variational Autoencoders (CVAEs) often suffer from "posterior collapse," where the model learns to ignore control signals. Furthermore, mapping attributes to a single continuous Gaussian distribution tends to conflate different attributes, leading to a lack of nuance. For example, a model might struggle to distinguish between the "content" of a restaurant review and the "polite style" in which it is delivered.
Methodology: The Focused-Variation Network
The Focused-Variation Network (FVN) builds upon the Vector-Quantised VAE (VQ-VAE) framework to address these issues through the use of disjoint discrete latent spaces.
Disjoint Codebooks
Instead of a single latent space, FVN maintains separate "codebooks" for content and style. These codebooks act as a discrete memory, storing specific ways that content and style were expressed in the training data. This discreteness prevents the "averaging" effect that makes many generative models produce bland, middle-of-the-road text.
Architecture and Training
During training, the model uses four encoders: two that process the raw text (text-to-content and text-to-style) and two that process the target attributes (content and style). At inference time, the model uses a learned frequency distribution to sample "codes" from the codebooks that are most relevant to the requested attributes. This allows for "focused" generation that is still varied. To ensure the generator follows the attributes, the model includes classification heads that perform "backward prediction," guessing the original content and style from the generated output.
Experimental Setup
The model was evaluated on two major benchmarks for task-oriented dialogue:
- PersonageNLG: A dataset containing over 88,000 examples annotated with both Content Meaning Representations (CMR) and five personality traits (e.g., agreeable, disagreeable, conscientious).
- E2E (End-to-End Challenge): A dataset of 42,000+ examples focused on restaurant descriptions, used to test FVN's ability to handle complex, lexicalized content.
Key Results
FVN achieves state-of-the-art performance across multiple dimensions of text generation:
- State-of-the-Art Quality: On the PersonageNLG dataset, FVN achieved a BLEU score of 0.965, significantly outperforming previous baselines.
- Perfect Style Control: In style evaluation, FVN achieved a near-perfect 1.0 F1 score, meaning it almost never failed to adopt the requested persona.
- Human-Level Fluency: In crowdsourced human evaluations, FVN’s fluency was rated at 2.81/3.0, nearly identical to ground-truth human text (2.9/3.0).
- Surpassing Humans in Style: For "agreeable" and "conscientious" personas, human judges actually preferred FVN’s generated text over the original human-written references.
Conclusion and Future Work
By decoupling content and style through discrete codebooks, FVN provides a robust framework for controllable text generation. While the current model relies on delexicalization for some tasks, future work will focus on moving toward fully lexicalized generation and applying disjoint codebooks to more complex structured data, such as knowledge graphs.
Summary updated