Abstract
Controllable Text Generation (CTG) requires a delicate balance between adhering to specific attributes and maintaining linguistic diversity. We present the Focused-Variation Network (FVN), a novel framework that utilizes disjoint, discrete latent spaces (codebooks) to decouple content from style. By sampling from focused distributions learned during training, FVN achieves state-of-the-art controllability and diversity. Evaluated on the PersonageNLG and E2E benchmarks, FVN significantly outperforms existing baselines, achieving a BLEU score of 0.965 and matching or exceeding human performance in conveying personality traits.
1. Introduction: The Challenge of Controllable Generation
Controllable Text Generation (CTG) aims to produce fluent text that strictly adheres to specified attributes, such as informational content (e.g., "The restaurant is Italian") and stylistic markers (e.g., "Speak like an extravert"). Despite its importance for dialogue systems, existing approaches face two major hurdles:
- The Controllability-Diversity Trade-off: Models often struggle to follow attributes while remaining diverse. Standard Language Models (LMs) are difficult to steer, while Conditional Variational Autoencoders (CVAEs) frequently suffer from "posterior collapse," where the model ignores latent variables, resulting in dull, repetitive outputs.
- Attribute Conflation: Most models utilize a single latent space for all attributes. This conflation makes it difficult for the model to distinguish between "content" and "style," leading to "leaky" generation where stylistic markers might overwrite factual content or vice versa.
2. Methodology: The Focused-Variation Network
The authors propose the Focused-Variation Network (FVN), which extends the Vector-Quantized VAE (VQ-VAE) framework into a conditional setting. The core innovation lies in its use of discrete latent representations to provide more robust control than continuous Gaussian distributions.
Key Architectural Components:
- Disjoint Codebooks: FVN maintains separate "codebooks" for content and style. These act as discrete memories storing different linguistic templates to realize specific attributes.
- Focused Variation Sampling: During inference, the model does not sample randomly. Instead, it analyzes the frequency distributions of codebook indices used during training for a specific attribute and samples from that "focused" distribution, mimicking human-like variety.
- Dual-Stream Encoding: The architecture employs four encoders: two mapping reference text to content/style latent spaces, and two mapping intended control factors (attributes) to latent vectors.
- Auxiliary Classification Heads: To ensure latent vectors capture the intended information, the model includes content and style decoders that perform backward prediction tasks during training.
3. Datasets and Experimental Setup
The model was rigorously evaluated on two major benchmarks for natural language generation in dialogue systems:
- PersonageNLG: A dataset of approximately 89,000 examples annotated with 8 content slots and 5 personality traits (agreeable, disagreeable, conscientious, unconscientious, and extravert).
- E2E (End-to-End Challenge): A dataset of 42,000 examples focusing on complex restaurant descriptions. The authors adapted FVN to treat different slot-value pairs as attributes to test content variation.
4. Key Insights and Results
The FVN model demonstrated significant improvements over previous state-of-the-art models such as TGEN and SLUG. By forcing content and style into disjoint spaces, the model can vary the "flavor" of a sentence without losing the underlying "facts."
- Automatic Metrics: On PersonageNLG, FVN achieved a BLEU score of 0.965, vastly outperforming the best baseline (0.559). It also achieved near-perfect scores (1.0) for both style and content correctness.
- Diversity: FVN maintained a high ratio of distinct n-grams, proving it avoids the trap of falling back on a single "safe" sentence for every attribute.
- Human Evaluation: In a significant finding, human evaluators judged FVN-generated text as equal to or better than ground truth 50.29% of the time. For "agreeable" and "conscientious" personalities, FVN actually outperformed human-written references in conveying the intended style.
5. Limitations and Future Directions
While FVN represents a major step forward, some limitations remain. The content codebook effectively "memorizes" training variations, which may hinder generalization to entirely new, out-of-domain content structures. Additionally, if training data contains missing values for certain attributes, the codebook may learn to omit those values during generation.
Future work could expand this architecture to more complex attributes, such as linguistic register, varying readability levels, or even multi-modal attributes, such as generating text based on visual image features.
Summary updated