Substance over Style: Document-Level Targeted Content Transfer

Allison Hegel, Sudha Rao, Asli Celikyilmaz, Bill Dolan

Abstract

Existing language models excel at writing from scratch, but many real-world scenarios require rewriting an existing document to fit a set of constraints. Although sentence-level rewriting has been fairly well-studied, little work has addressed the challenge of rewriting an entire document coherently. In this work, we introduce the task of document-level targeted content transfer and address it in the recipe domain, with a recipe as the document and a dietary restriction (such as vegan or dairy-free) as the targeted constraint. We propose a novel model for this task based on the generative pre-trained language model (GPT-2) and train on a large number of roughly-aligned recipe pairs. Both automatic and human evaluations show that our model out-performs existing methods by generating coherent and diverse rewrites that obey the constraint while remaining close to the original document.

Introduction: Beyond Stylistic Shifts

Most research in text style transfer focuses on stylistic shifts—changing a sentence from formal to informal or positive to negative. However, real-world rewriting often requires changing the substance of a document.

A major gap in current research is document-level coherence. For example, turning a traditional beef stew recipe into a vegan one requires more than just swapping "beef" for "tofu" in a single sentence. One might need to adjust cooking times (tofu doesn't require hours of braising), change the broth, or swap out garnishes. These changes are interdependent and span the entire document. Current models that rewrite sentence-by-sentence fail to maintain this "global" logic, leading to incoherent or "hallucinated" instructions.

Figure 1: Comparison between an original hot cocoa recipe and a rewritten vegan version, highlighting substantive changes beyond simple word swaps.

Methodology: Context and Prompts

The problem is framed as a sequence-to-sequence task using GPT-2 (medium) as the backbone. The researchers proposed two main model variants to ensure global consistency:

  1. Contextual Rewriter: This model generates the target recipe one step at a time. For each step, it "sees" the entire source recipe (title, ingredients, and all steps) plus all the target steps it has generated so far. This allows the model to maintain document-level context.
  2. Contextual Rewriter + Ingredient Prompt: To prevent the model from straying too far from the original recipe, the authors added a "prompt" for each step. This prompt is a list of ingredients that should be used in that specific step. At test time, these prompts are generated either by a rule-based system (using a food substitution guide) or a secondary GPT-2 model.
Figure 3: A comparison of the model architectures, showing how the Ingredient Prompt guides the Contextual Rewriter to maintain focus on specific ingredients for each step.

Dataset and Training

The authors compiled a massive dataset of 1.25 million recipes scraped from the web. To create a "parallel" dataset for training, they used an unsupervised alignment algorithm to group recipes by dish name (e.g., all "Hot Cocoa" recipes) and pair a version that violated a constraint (non-vegan) with a version that obeyed it (vegan). The study focused on seven dietary restrictions: dairy-free, nut-free, egg-free, vegan, vegetarian, alcohol-free, and fish-free.

Key Results

The Contextual Rewriter + Rule Prompt emerged as the superior model across both automatic and human evaluations:

  • Dietary Adherence: It achieved 99.5% accuracy in following dietary constraints, significantly outperforming retrieval-based baselines (~93.4%).
  • Source Fidelity: Using ROUGE scores, the model demonstrated it stayed much closer to the original recipe's logic than controllable generation models like PPLM or CTRL, which often ignored the source recipe entirely.
  • Human Preference: Human judges preferred the model's rewrites over simpler rule-based substitutions. The model was specifically praised for making "coordinated changes"—such as adding nutritional yeast for flavor in a dairy-free recipe rather than simply deleting the cheese.

Conclusion and Future Directions

While the model successfully handles substantive document-level transfer, it still lacks "physical common sense." It occasionally suggests impossible actions or illogical cooking times. Future work should focus on grounding language models in physical reality to prevent these common-sense blunders and bridge the gap between AI-generated content and expert human-authored recipes.

Summary updated

Abstract

Existing language models excel at writing from scratch, but many real-world scenarios require rewriting an existing document to fit a set of constraints. Although sentence-level rewriting has been fairly well-studied, little work has addressed the challenge of rewriting an entire document coherently. In this work, we introduce the task of document-level targeted content transfer and address it in the recipe domain, with a recipe as the document and a dietary restriction (such as vegan or dairy-free) as the targeted constraint. We propose a novel model for this task based on the generative pre-trained language model (GPT-2) and train on a large number of roughly-aligned recipe pairs (https://github.com/microsoft/document-level-targeted-content-transfer). Both automatic and human evaluations show that our model out-performs existing methods by generating coherent and diverse rewrites that obey the constraint while remaining close to the original document. Finally, we analyze our model's rewrites to assess progress toward the goal of making language generation more attuned to constraints that are substantive rather than stylistic.
BibTeX
@inproceedings{hegel2020substance,
  author = {Allison Hegel and Sudha Rao and Asli Celikyilmaz and Bill Dolan},
  title = {{Substance over Style: Document-Level Targeted Content Transfer}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=wM6QVgcAxad},
}
Figure