TL;DR
This paper introduces SAIF, a novel multimodal deep learning architecture designed to detect when a user is correcting a previous command given to a personal assistant. By leveraging both voice cues (like tone and emphasis) and text transcripts, SAIF aims to help intelligent agents learn from their mistakes and improve user interaction, outperforming existing methods on a newly released dataset.
What Problem Does This Paper Address?
As virtual assistants become more integrated into our lives, their ability to understand and correctly execute commands is crucial. However, these agents often misinterpret user requests, leading to errors. While users might try to correct these mistakes by rephrasing their commands, current assistants typically don't learn from these implicit corrections. This paper tackles the challenge of automatically detecting these user corrections, distinguishing between genuine command corrections and errors arising from faulty Automatic Speech Recognition (ASR). The goal is to equip personal assistants with the intelligence to recognize and act upon these user-initiated corrections, thereby improving their robustness and learning capabilities.
Methods
The core contribution is the SAIF (Socially Aware personal assistant Implicit Feedback correction detector) architecture. SAIF is a multimodal deep learning model that processes pairs of consecutive user commands. It takes two primary inputs:
- Transcripts: The text of the user's commands. SAIF uses a pre-trained Skip-Thoughts encoder to convert these transcripts into fixed-length vectors. It then computes the component-wise product and absolute difference of these vectors, concatenating them.
- Voice: The audio of the user's commands. SAIF employs a pre-trained emotion recognition model (from DataFlair) to encode voice commands into vectors. It also incorporates a Voice Activity Detection (VAD) feature that measures the difference in speech duration between consecutive commands.
These transcript and voice representations are then fed into separate hidden layers. The outputs of these layers are concatenated with an additional input: an indicator of whether the agent successfully executed the previous command. This combined vector is then processed through further hidden layers, culminating in a final layer that outputs probabilities for three classes: "new command" (no correction), "command correction" (user is correcting a misunderstanding of the command's intent), and "ASR correction" (user is correcting a misinterpretation by the speech recognition system).
Datasets
The authors introduce a newly collected dataset for this research. This dataset comprises 2540 pairs of consecutive spoken commands given to a social agent named LIA. The data was collected from 20 users (11 male, 9 female) aged 18-62. Each command pair includes the original voice file, the ASR transcript, and an indicator of the agent's execution success. Crucially, the dataset is manually labeled with three categories: "new command," "command correction," and "ASR correction." This dataset is unique in its focus on multimodal correction detection and its specific labeling scheme.
Key Insights and Ideas
- Multimodality is Key: The paper strongly emphasizes that combining voice and transcript information significantly improves correction detection accuracy. Voice cues like tone, speed, and emphasis provide valuable context that text alone misses, especially for ASR errors.
- Distinguishing Correction Types: Differentiating between "command corrections" and "ASR corrections" is vital. ASR corrections require improving the speech recognition system, while command corrections necessitate the agent learning new command aliases or understanding user intent better.
- Implicit Feedback is Powerful: The research highlights the potential of using implicit user feedback (like corrections) to make agents more adaptive and intelligent, moving beyond simple command execution.
- Dataset Importance: The creation and release of a labeled multimodal dataset are critical for advancing research in this area, as prior benchmarks were either text-only or lacked the specific focus on correction detection.
Key Results
SAIF achieved an average test accuracy of 0.818 on the custom dataset. This is a significant improvement over a baseline accuracy of 0.68 that would be achieved by simply guessing "new command" for every instance. The model also achieved F1 scores of 0.69 for "command correction" and 0.344 for "ASR correction."
When compared to various baselines, including transcript-only and voice-only approaches, SAIF consistently outperformed them. For instance, a transcript-only approach using Skip-Thoughts achieved an accuracy of 0.742, and voice-only methods performed poorly, often guessing "new command" almost exclusively. Even architectures incorporating BERT did not match SAIF's performance.
Limitations and Future Work
- Dataset Size: While 2540 pairs is a good start, the authors acknowledge that the voice pre-training model was trained on a much larger dataset (74 million sentences for Skip-Thoughts vs. 7356 voice files for the emotion model). They suggest that a larger voice dataset for pre-training could further improve SAIF's performance.
- Consecutive Commands Only: SAIF currently only considers pairs of consecutive commands. The authors propose extending this to analyze three or more consecutive commands, which could provide richer context for detecting complex corrections.
- High Precision Requirement: For practical deployment, SAIF needs very high precision. Incorrectly identifying a command as a correction could lead to unintended undoing of actions or learning incorrect command aliases, negatively impacting user experience. Future work might involve explicit user confirmation for suspected corrections or requiring multiple instances of a correction before acting on it.
- Generalization: While SAIF aims to be agent-independent, further testing on diverse agents and interaction scenarios would be beneficial.
Summary updated