Introduction
Detecting a user's "Level of Interest" (LoI) is a critical component of affective computing, particularly for enhancing human-computer interaction in applications like AI tutors or automated customer service. While traditional systems have focused primarily on acoustic features—such as pitch, energy, and tempo—this research, presented at Interspeech 2010, investigates the integration of lexical content. A central challenge in this domain is the inherent noise in Automatic Speech Recognition (ASR). This paper explores whether lexical features can provide a robust signal even when ASR output contains significant errors.
Methodology
The researchers proposed a multi-level fusion framework that integrates acoustic, contextual, and lexical evidence to predict a speaker's interest level.
The system architecture consists of three primary components:
- Acoustic Evidence: Standard acoustic features were processed using classifiers such as Random Subspace (baseline) and Gaussian Processes.
- Contextual Evidence: To ensure temporal consistency, the model incorporates the predicted interest level of the previous turn ($P_{t-1}$) to smooth predictions for the current turn ($P_t$).
- Lexical Evidence: A "bag-of-words" model was employed, treating both words and non-linguistic vocalizations (e.g., laughter, sighs, pauses) as features. Three weighting schemes were evaluated: Term Frequency (TF), Subjectivity (SUBJ) based on emotional intensity, and LoI Weighting based on average interest levels in training data.
These components were fused using weighted linear interpolation, with weights optimized on a development set.
Experimental Setup
The study utilized the TUM AVIC (Audiovisual Interest Corpus), which contains recordings of spoken dialogs labeled with LoI. To simulate real-world conditions with varying ASR quality, the researchers used the Sphinx-3 system, supplemented with training data from the TIMIT and Fisher corpora, to generate transcripts with different Word Error Rates (WER).
Results and Discussion
The experimental results highlight the significant impact of lexical information on system performance. The acoustic-only baseline achieved a Correlation Coefficient (CC) of 0.604. Integrating lexical data from human transcripts pushed the CC to 0.627, representing the single largest performance gain among the fusion components.
A key finding is the resilience of lexical features to ASR noise. As shown in the figure below, while performance naturally declines as WER increases, the fused system remains remarkably informative even at high error rates, staying above the acoustic-only baseline even at 74% WER.
Specifically, when moving from perfect transcripts to ASR output with a 34% WER, the lexical model's CC only decreased from 0.507 to 0.475. Even at a 74% WER, the lexical features alone maintained a CC of 0.406. This suggests that the "gist" of a speaker's interest level is often preserved despite transcription errors. Furthermore, non-linguistic cues like laughter and the overall length of the turn (LEN) were identified as powerful indicators of engagement.
However, the study also noted limitations. The training data was skewed toward neutral interest levels (0.4 to 0.8 LoI), making it difficult to predict extreme states. Additionally, high ASR error rates on the final blind test set (~70%) limited the observed gains from lexical fusion compared to the development set.
Conclusion
This research demonstrates that lexical content is a powerful and resilient signal for sensing Level of Interest in spoken dialog. By fusing acoustic cues with lexical and contextual evidence, the proposed system achieves superior performance compared to acoustic-only baselines. Crucially, the effectiveness of lexical features persists even under significant ASR noise, proving their viability for real-world affective computing applications. Future work will focus on more robust word selection and advanced modeling techniques to further mitigate the impact of speech recognition errors.
Summary updated