Introduction
Virtual assistants frequently sound robotic because they repeat user messages verbatim instead of adjusting pronouns or syntax. For instance, if a user says, "Tell Alice I’ll meet her in 10 minutes," a naive system might send "I’ll meet her in 10 minutes" to Alice, rather than the more natural "I'll meet you in 10 minutes." This paper introduces the Message Content Rephrasing (MCR) task and a corresponding dataset to address this limitation.
The Challenge of Naturalness
The MCR task requires a "minimal syntactic rewrite"—changing pronouns or turning statements into questions while maintaining the original meaning. Unlike standard paraphrasing, which might change style or wording, MCR is a precise edit based on the recipient's perspective.
The authors released the MCR dataset, consisting of 3,000 pairs of original queries and rephrased targets. The dataset is split into EXACT (verbatim content) and REPHRASE (requiring pronoun or verb shifts) categories. Rephrasing typically involves pronoun shifts (8%), turning statements into questions (13%), or both (22%).
Modeling Approaches
The study explores several modeling directions to solve the MCR task:
- LSTM Seq2Seq with Pointer-Generator: A baseline model using GloVe and ELMo embeddings. It includes a "pointer-generator" mechanism to either generate new words or copy directly from the input, which is essential for preserving names and specific details.
- BART with Explicit Copying: The authors utilize BART, a powerful pre-trained transformer. To prevent "hallucinations" or incorrect changes to proper nouns, they added an explicit copying head during fine-tuning, initialized from the transformer's existing attention heads.
- LaserTagger (Edit-based): This model treats the task as a tagging problem (Keep, Delete, or Add phrase) rather than generating text from scratch.
- Knowledge Distillation (KD): To make the system fast enough for real-world use, the authors used the large BART model as a "teacher" to train a much smaller, production-ready LSTM "student" model.
Experimental Results
The evaluation focused on accuracy metrics like Exact Match (EM) and SARI (a metric for text editing).
Key findings include:
- Best Overall Performer: BART with explicit copying achieved the highest scores, reaching 89.3% EM and a SARI score of 86.5.
- Copying is Essential: Vanilla BART had a high "Copy Related" error rate (24%), proving that even large language models require explicit architectural constraints to remain faithful to the source text.
- Efficiency via Distillation: The distilled LSTM is 10x faster on a CPU than BART while maintaining a significant portion of the performance gains (85.4% EM), making it viable for on-device deployment.
Conclusion
While large models like BART excel at rephrasing, the addition of an explicit copy mechanism and the use of knowledge distillation are critical for creating systems that are both accurate and efficient. Future work will need to address more complex social contexts, such as politeness strategies, and expand the dataset to cover a broader range of linguistic edge cases.
Summary updated