A Salience Driven Approach to Robust Input Interpretation in Multimodal Conversational Systems

Joyce Y. Chai, Shaolin QuEMNLP 2005 · OpenReview

TL;DR

This paper introduces a "salience-driven" approach to improve how multimodal conversational systems understand user input. By using deictic gestures (like pointing) to highlight important parts of the visual interface, the system can better predict what the user will say next, leading to more accurate speech understanding and concept identification.

What Problem Does This Paper Address?

Multimodal conversational systems, which combine speech with other inputs like gestures, aim for more natural human-computer interaction. However, interpreting this combined input is challenging due to limitations in speech recognition and language understanding. Existing methods often fuse information from different modalities late in the process, potentially losing crucial, low-probability information. This paper tackles the problem of how to leverage information from one modality (gestures) earlier to directly influence the processing of another (speech), specifically to improve spoken language understanding before the final fusion stage.

Methods

The core of the proposed approach is a "salience-driven" method. It's based on the idea that when a user points to or circles something on a screen, that object or area becomes "salient" – it's what the user is currently focused on. This salience information, derived from deictic gestures, is then used to "prime" the language model for spoken input.

Here's a breakdown:

  • Salience Modeling: Gestures are used to calculate a "salience distribution" over entities in the application's "physical world representation" (which includes domain knowledge like objects, properties, and tasks). This distribution indicates which entities are currently most important to the user. The salience of an entity at a given time is influenced by the temporal proximity of preceding gestures.
  • Primed Language Model: Instead of using a standard language model, they use a "primed" class-based bigram model. The salience distribution dynamically adjusts the word class probabilities. This means the model is more likely to predict words related to the currently salient entities, effectively biasing the language model towards more relevant hypotheses.
  • Early Integration: Crucially, this salience information is used to influence the language model before multimodal fusion, specifically during the post-processing of recognized speech hypotheses. This allows potentially low-probability but contextually relevant information to be considered.

Datasets

The paper uses data collected from previous user studies where participants interacted with a multimodal interface using both speech and deictic gestures to find information about real estate properties. The study involved eleven subjects, and the data consisted of 226 user inputs (turns). The authors mention that the data was annotated in terms of semantic information and intended objects of interest, which was used for training the domain grammar and estimating probabilities.

Key Insights and Ideas

  • Gestures as Predictive Cues: The observation that deictic gestures often precede spoken utterances is key. This temporal advantage allows gestures to act as an early warning system, anticipating the user's communicative intent.
  • Salience as a Bridge: Salience, derived from visual cues like gestures, acts as a powerful bridge between modalities. It connects the user's focus in the visual world to their linguistic expression.
  • Priming for Robustness: Directly influencing language models with salience information offers a more robust way to handle recognition uncertainties. Instead of just filtering out errors, it actively guides the system towards more probable interpretations based on context.
  • Beyond Reference Resolution: The paper extends the concept of salience beyond just resolving referring expressions (like pronouns) to influencing the entire language understanding process.

Key Results

The experimental results demonstrate the effectiveness of the salience-driven approach:

  • Reduced Word Error Rate (WER): The salience-based post-processing significantly reduced the word error rate compared to a baseline. The average WER reduction was about 12%, with a statistical significance (t = 4.75, p < 0.001).
  • Improved Concept Identification: The approach also led to better concept identification. On average, precision increased from 80.3% to 84.6%, and recall increased from 75.7% to 83.8%, resulting in a higher F-measure (83.8% vs. 84.2%).
  • Qualitative Improvement: Examples provided show how the salience-driven approach corrects misrecognized words (e.g., "publisher" to "population," "great" to "gray") that were likely due to acoustic or language model ambiguities.

Limitations and Future Work

  • Limited Domain and Vocabulary: The current evaluation was conducted on a small-scale domain with a limited vocabulary. The authors acknowledge that this demonstrates the potential but needs further validation on more complex domains.
  • Gesture-Centric Salience: The current salience modeling primarily relies on gesture information.
  • Future Directions: The authors plan to explore more complex domains with richer concepts and relations. They also aim to extend salience modeling to include other modalities like eye gaze and conversation context. Investigating how to combine information from multiple sources for salience modeling and applying these models at earlier processing stages are also key future research directions.

Summary updated

Abstract

To improve the robustness in multimodal input interpretation, this paper presents a new salience driven approach. This approach is based on the observation that, during multimodal conversation, information from deictic gestures (e.g., point or circle) on a graphical display can signal a part of the physical world (i.e., representation of the domain and task) of the application which is salient during the communication. This salient part of the physical world will prime what users tend to communicate in speech and in turn can be used to constrain hypotheses for spoken language understanding, thus improving overall input interpretation. Our experimental results have indicated the potential of this approach in reducing word error rate and improving concept identification in multimodal conversation.
BibTeX
@inproceedings{chai2005salience,
  author = {Joyce Y. Chai and Shaolin Qu},
  title = {{A Salience Driven Approach to Robust Input Interpretation in Multimodal Conversational Systems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
  url = {https://openreview.net/forum?id=HkZa2GbOWB},
}
Figure