Acoustic-linguistic recognition of interest in speech with bottleneck-BLSTM nets

Martin Wöllmer, Felix Weninger, Florian Eyben, Björn SchullerInterspeech 2011

TL;DR

This paper introduces a novel approach to automatically recognizing a speaker's level of interest in natural conversations by combining acoustic and linguistic speech features. They achieve state-of-the-art results by employing a "bottleneck-BLSTM" neural network architecture, which efficiently learns long-range contextual information from speech.

What Problem Does This Paper Address?

Understanding a person's interest level in a conversation is crucial for many human-computer interaction applications, such as personalized sales systems, virtual assistants, and educational tools. While previous research has explored this, comparing different approaches has been challenging due to a lack of standardized datasets and evaluation methods. The Interspeech 2010 Paralinguistic Challenge aimed to address this by defining a specific task for interest recognition using the TUM AVIC corpus, which contains spontaneous commercial presentations. The challenge was to predict a speaker's interest level from speech signals, often requiring the model to capture subtle, evolving cues over time.

Methods

The core of this paper's contribution lies in its novel neural network architecture: bottleneck-BLSTM nets.

  • Bidirectional Long Short-Term Memory (BLSTM): This is an extension of standard Recurrent Neural Networks (RNNs) that can process sequences in both forward and backward directions. This is particularly useful for interest recognition, as a person's current interest might be influenced by both past and future conversational context. BLSTMs are adept at learning long-range dependencies, overcoming the "vanishing gradient problem" that plagues traditional RNNs.
  • Bottleneck Architecture: This refers to a specific structure within the neural network where a narrow "bottleneck" layer is placed in the middle of multiple hidden layers. This forces the network to learn a compressed, low-dimensional representation of the input features. In this paper, the bottleneck layer is used for context-sensitive feature compression, effectively reducing dimensionality and decorrelating features implicitly.
  • Acoustic and Linguistic Features: The system integrates two types of information:
  • Acoustic Features: Standard Mel-Frequency Cepstral Coefficients (MFCCs) are extracted using the openSMILE toolkit, along with statistical functionals to capture temporal dynamics.
  • Linguistic Features: These are derived from a sophisticated multi-stream BLSTM-HMM speech recognizer. This recognizer uses BLSTM for context-sensitive phoneme prediction, which is then combined with MFCCs in an HMM to produce linguistic information.
  • Fusion: The acoustic and linguistic features are fused early in the process. The bottleneck-BLSTM network then processes this combined feature set to predict the level of interest.

Datasets

  • TUM AVIC Corpus: This is the primary dataset used for training, development, and testing. It was also used for the Interspeech 2010 Paralinguistic Challenge. The corpus contains spontaneous speech from commercial presentations, with "level of interest" (LOI) annotated on a five-point scale (disinterest to curiosity), later mapped to a continuous scale from -1 to +1. The dataset is split into speaker-independent training, development, and test sets.

Key Insights and Ideas

  • Context is King: The paper strongly emphasizes that capturing long-range temporal context is crucial for accurate interest recognition. BLSTM networks, by their nature, excel at this.
  • Bottleneck for Efficiency and Effectiveness: The bottleneck architecture isn't just about reducing computation; it actively contributes to learning a more robust and compressed feature representation, leading to better performance.
  • Synergy of Acoustic and Linguistic Cues: Combining acoustic features (how something is said) with linguistic features (what is said, derived from ASR) yields significantly better results than using either alone.
  • Bidirectional Advantage: While unidirectional LSTMs are effective, the bidirectional approach (BLSTM) further improves performance, suggesting that future contextual information is indeed valuable for interest prediction.
  • Bottleneck-BLSTM Outperforms Simpler Models: The proposed bottleneck-BLSTM architecture consistently outperforms simpler BLSTM or RNN models with a single hidden layer, highlighting the benefits of the layered bottleneck structure.

Key Results

The paper reports state-of-the-art results on the Interspeech 2010 Paralinguistic Challenge task.

  • The best performing system was a bottleneck-BLSTM network processing acoustic and linguistic features, achieving a cross-correlation (CC) of 0.504.
  • This result significantly outperforms previous baseline methods for the challenge, including REP-Trees (CC of 0.421/0.423), SVM (CC of 0.428), and GMM (CC of 0.390).
  • The paper also shows that incorporating linguistic features consistently improves cross-correlation across various network architectures.
  • Bottleneck architectures generally outperformed their non-bottleneck counterparts.

Limitations and Future Work

  • Limitations:
    • The paper notes that using only the training set for training predictors did not yield satisfying results, indicating that the proposed neural network architectures require a substantial amount of training data for good generalization.
  • The current system uses a Bag-of-Words (BoW) approach for linguistic features, which might not fully capture the nuances of sentence structure or semantic meaning.
  • Future Work:
    • Multi-task Learning: The authors suggest exploring multi-task learning to simultaneously predict interest along with other emotional dimensions like valence and arousal.
  • Framewise Modeling: Instead of using statistical functionals on supra-segmental acoustic features, they propose investigating framewise modeling of interest.
  • Improved Linguistic Features: Future work could explore more sophisticated methods for extracting linguistic information beyond BoW, potentially leveraging advanced natural language processing techniques.

Summary updated

BibTeX
@inproceedings{wollmer2011acoustic,
  author = {Martin Wöllmer and Felix Weninger and Florian Eyben and Björn Schuller},
  title = {{Acoustic-linguistic recognition of interest in speech with bottleneck-BLSTM nets}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure