Converting the Point of View of Message Spoken to Virtual Assistants

Gunhee Lee, Vera Zu, Sai Srujana Buddi, Dennis Liang, Purva Kulkarni, Jack G. M. FitzgeraldEMNLP 2020 · DOI

Virtual assistants (VAs) often act as literal conduits, relaying messages exactly as dictated. For instance, a command like "Tell Bob I love him" typically results in Bob receiving the text "I love him." This research introduces a system designed to automatically convert the Point of View (POV) of voice messages, enabling the VA to act as a natural intermediary. Instead of verbatim relay, the system produces more natural outputs such as "Joe says he loves you."

The POV Mismatch Problem

Current VA systems primarily use "snipping," where the assistant identifies the message content and relays it without modification. This leads to two significant issues:

  1. POV Mismatch: A request like "Ask Bob if he’s coming" results in Bob receiving "he’s coming" instead of the intended "Are you coming?"
  2. Unnatural Interaction: A true intermediary should frame the message appropriately (e.g., "Joe asks if you are coming").

Solving this requires addressing complex linguistic challenges, including co-reference resolution (mapping "he" to "Bob"), pronoun shifting ("I" to "he"), verb agreement adjustments, and structural changes like subject-auxiliary inversion for questions.

Methodology

The researchers evaluated two primary approaches: a rule-based system and various Neural Machine Translation (NMT) architectures.

Rule-Based Model

This approach utilizes a linear classifier to categorize messages into four types: Statements, Yes/No Questions, Wh-Questions, and Requests. Following classification, the system employs Part-of-Speech (POS) tagging and constituency parsing to apply specific grammatical transformation rules.

Neural Models

The task was also framed as a translation problem—converting "Sender-speak" to "VA-speak." Three architectures were tested:

  • LSTM-LSTM: A standard sequence-to-sequence model with attention.
  • CopyNet: A specialized model capable of "copying" tokens from the input, which is vital for preserving names and rare entities.
  • T5 (Text-to-Text Transfer Transformer): A large-scale pre-trained transformer fine-tuned for POV conversion.
System architecture for POV conversion, illustrating the flow from input utterance through NLU and transformation components to the final output.

Dataset and Evaluation

To support this research, a new benchmark dataset of 46,565 crowd-sourced samples was created. Utterances were collected from users and manually converted into ideal "VA-speak" outputs. To improve generalization, contact names were replaced with special tokens (e.g., @CN@).

Key Results and Insights

The models were evaluated on Faithfulness (using BLEU and METEOR scores) and Naturalness (measured via Relative Perplexity).

Model Params BLEU METEOR Rel. Perplexity
Rule-based - 46.6 72.3 0.918
CopyNet 5.3M 63.1 82.0 1.54
T5 220M 63.8 83.0 1.42
Tsfmr-CopyNet 5.9M 63.7 82.8 1.59

While T5 achieved the highest faithfulness scores, CopyNet variants proved to be more efficient and produced more "natural" sounding outputs. Specifically, CopyNet is approximately 37 times smaller than T5, making it a more viable candidate for on-device deployment where latency is critical.

Relative perplexity across training epochs for LSTM-LSTM, CopyNet, and T5 models, showing convergence and naturalness trends.

The study also highlighted a "Naturalness Paradox": models often scored higher on naturalness than human ground truth because they favored consistent, safe "carrier phrases" (e.g., "John is asking...") over the more varied phrasing of human contributors.

Conclusion and Future Directions

The research demonstrates that specialized neural architectures like CopyNet can effectively handle POV conversion with high faithfulness and superior efficiency compared to massive language models. Future work will focus on addressing Out-of-Vocabulary (OOV) issues using pre-trained embeddings and extending the system to handle multi-turn "messaging chains" for better contextual pronoun resolution.

Summary updated

BibTeX
@inproceedings{lee2020converting,
  author = {Gunhee Lee and Vera Zu and Sai Srujana Buddi and Dennis Liang and Purva Kulkarni and Jack G. M. Fitzgerald},
  title = {{Converting the Point of View of Message Spoken to Virtual Assistants}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.15},
}
Figure