Abstract
Allison Hegel, Sudha Rao, Asli Celikyilmaz, Bill Dolan. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Most research in text style transfer focuses on stylistic attributes like sentiment or formality, usually at the single-sentence level. However, real-world rewriting often requires substantive changes across an entire document. This paper introduces document-level targeted content transfer, a task where a model must rewrite an entire document to satisfy a specific constraint—such as making a recipe vegan—while maintaining the original's structure and coherence.
The Challenge of Substantive Transfer
Content transfer is significantly more complex than style transfer because it requires domain-specific "world knowledge." For instance, if you want to make a non-vegan hot cocoa recipe vegan, you cannot simply replace "milk" with "soy milk." You might need to adjust cooking temperatures, simmering times, or replace secondary ingredients like heavy cream. These changes are interdependent: a modification in the first step affects the requirements of the third. The challenge lies in building a model that understands these document-wide dependencies while remaining loyal to the original text's intent.
Methodology: Contextual Rewriting
The authors frame the problem as a sequence-to-sequence task using GPT-2 (medium) as the backbone. They propose two primary model variants:
- Contextual Rewriter: The model generates the target document one step at a time. For each step, it receives the entire source document (title, ingredients, and all steps) plus any target steps it has already generated. This allows the model to maintain global context and local consistency.
- Contextual Rewriter + Ingredient Prompt: To further guide the model, the authors provide a "prompt" for each step consisting of the specific ingredients that should be used. These prompts act as a "storyline," helping the model stay on track with the original structure while obeying the target constraint.
Data and Unsupervised Alignment
To train these models without a "gold" dataset of manual rewrites, the authors developed an unsupervised alignment algorithm. They utilized a massive crawl of approximately 1.25 million recipes, pairing different versions of the same dish where one violated a dietary constraint (e.g., dairy-free, nut-free, vegan) and the other satisfied it. This resulted in hundreds of thousands of noisy parallel pairs, providing enough signal for the transformer model to learn patterns of substitution and technical adjustment.
Key Results and Insights
The proposed models were compared against several baselines, including retrieval-based methods and controllable generation models like PPLM and CTRL.
- Dietary Adherence: The models achieved over 99% adherence to dietary constraints, significantly outperforming standard controllable generation models.
- Human Evaluation: Human judges preferred the "Contextual Rewriter + Rule Prompt" output over all other baselines, rating it 4.60/5.0 for overall quality.
- Diversity: The contextual rewriter maintained high trigram diversity (0.674), indicating it produced rich, varied instructions rather than repetitive text.
- The Human Gap: Despite strong performance, a gap remains between model outputs and "gold" rewrites by experienced cooks. Humans achieved a ROUGE score of 77.08 relative to the source, while the best model reached 35.44, suggesting room for improvement in capturing fine-grained nuances.
Conclusion
By fine-tuning GPT-2 with document-level context and ingredient prompts, the authors demonstrate that models can learn to make substantive, interdependent changes. While currently focused on recipes, this framework has potential applications in technical documentation and educational content adaptation. Future work will need to address the model's lack of physical grounding to avoid illogical cooking instructions.
Summary updated