Controlling Text Edition by Changing Answers of Specific Questions

Lei Sha, Patrick Hohenecker, Thomas Lukasiewicz

TL;DR

The paper introduces Controllable Text Edition (CTE), a task where a model minimally edits a document so that the answer to a specific question changes to a new target answer. Because parallel training data of edited documents is scarce, the authors propose Select-Mask-Generate (SMG)—an unsupervised framework that locates question-relevant text, masks it out, and dynamically generates fluent replacements to fit the new answer.

What Problem Does This Paper Address?

In many real-world scenarios—such as revising legal contracts or updating news articles—human editors must update specific facts without rewriting the entire text. Today’s NLP models typically approach text editing either through full-sequence regeneration (which can hallucinate or alter unrelated details) or simple string replacement (which fails when changes are non-contiguous, grammatically dependent, or not exact substrings).

The authors formalize this challenge as Controllable Text Edition (CTE): given an original document $D$, a question $Q$, and a new target answer $A'$, generate a minimally modified document $D'$ whose answer to $Q$ is now $A'$. The key technical hurdle is training models to know where and how to edit without having access to large-scale parallel datasets of before-and-after document revisions.

Methods

To circumvent the lack of parallel revision data, the authors introduce Select-Mask-Generate (SMG), which trains purely on single document–question–answer triples $\langle D, Q, A \rangle$:

  1. Selector Network: A BiLSTM with Gumbel-Softmax sampling predicts a binary mask $M$ over $D$, isolating tokens related to the original answer $A$.
  2. Answer Reconstruction: An answer decoder reconstructs the original answer $A$ solely from the masked-out tokens, ensuring the selector isolates necessary and sufficient information.
  3. Context-Aware Partial Decoder: Instead of regenerating the entire document, an LSTM decoder reads the unmasked context template and switches into a "writing mode" only when encountering masked blanks.
  4. A copy mechanism allows the decoder to copy tokens directly from the context and the answer.
  5. The model learns an End-of-Answer (EOA) indicator tag to predict exactly when to stop filling a blank and resume copying from the template.
  6. Inference: During inference, the selector masks the original text, and the decoder fills the gaps conditioned on the new answer $A'$ rather than $A$.

Datasets

  • WIKIBIOCTE: The authors constructed this dataset by adapting the table-to-text benchmark WIKIBIO. Infobox attributes (e.g., Occupation, Nationality) serve as questions $Q$, the infobox values serve as answers $A$, and the Wikipedia biography lead paragraph acts as $D$.
  • Training & Dev: Filtered to 26 common fields, yielding 141k training triples and 17.7k validation triples.
  • Evaluation Test Set: ~1,000 manually annotated examples $\langle D, Q, A, A', D' \rangle$, where linguists provided human-edited target texts $D'$ reflecting randomized alternative answers $A'$.

Key Insights and Ideas

  • Editing via Partial Generation: Full text regeneration via Seq2Seq models frequently contaminates the output—models often blend the old and new facts together (e.g., combining "gymnast" and "basketball" into "aerobic gymnast basketball"). Restricting generation exclusively to masked spans prevents hallucinations in unrelated context.
  • Answer Information as an Information Bottleneck: By forcing an auxiliary decoder to reconstruct the gold answer using only the selected tokens, the selector learns to isolate answer-bearing phrases without token-level supervision.
  • Dynamic Variable-Length Infilling: Unlike standard masked language models or fixed-slot infilling methods, the decoder uses the learned EOA token to fill blanks with arbitrary-length spans in a single autoregressive pass.

Key Results

Evaluating on the WIKIBIOCTE test set, SMG significantly outperformed standard sequence-to-sequence (Seq2Seq) baselines:

  • Targeted Revision (diff-BLEU ratio): SMG achieved 59.8% with predicted templates (and 62.5% with gold templates) compared to Seq2Seq's 21.3%, showing a vastly superior ability to rewrite the specific differences between documents.
  • iBLEU (penalizing pure copying): SMG scored 8.94 (predicted) vs. 5.63 (Seq2Seq).
  • Human Evaluation:
  • Correctness: Human judges found SMG successfully updated the factual answer 76.9% of the time (80.2% with gold templates) versus 73.5% for Seq2Seq.
  • Selection Performance: The selector achieved an answer $F_1$ score of 0.68 (compared to 0.55 for Seq2Seq attention-based selection).

Limitations and Future Work

  • Boundary Fluency: Partial generation can result in grammatical seams where the generated blank meets the fixed context. Consequently, Seq2Seq achieved better perplexity (198 vs. 373) and human fluency (4.56 vs. 4.32 on a 5-point scale).
  • Selector Precision: If the selector misses an answer-related token (e.g., leaving "gymnastic coach" while changing the sport elsewhere), inconsistent statements can remain in the document.
  • Generalization: The approach relies on question-answer extractions typical of structured infoboxes. Extending CTE to complex documents requiring multi-hop reasoning or cross-sentence discourse shifts remains an open challenge.

Summary updated

Abstract

In this paper, we introduce the new task of controllable text edition, in which we take as input a long text, a question, and a target answer, and the output is a minimally modified text, so that it fits the target answer. This task is very important in many situations, such as changing some conditions, consequences, or properties in a legal document, or changing some key information of an event in a news text. This is very challenging, as it is hard to obtain a parallel corpus for training, and we need to first find all text positions that should be changed and then decide how to change them. We constructed the new dataset WikiBioCTE for this task based on the existing dataset WikiBio (originally created for table-to-text generation). We use WikiBioCTE for training, and manually labeled a test set for testing. We also propose novel evaluation metrics and a novel method for solving the new task. Experimental results on the test set show that our proposed method is a good fit for this novel NLP task.
BibTeX
@inproceedings{sha2021controlling,
  author = {Lei Sha and Patrick Hohenecker and Thomas Lukasiewicz},
  title = {{Controlling Text Edition by Changing Answers of Specific Questions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=IF2JN-uMct},
}

Loading related content...

Figure