Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM

Ziqian Luo, Hua Xu, Feiyang ChenarXiv 2018 · OpenReview

TL;DR

This paper introduces AFF-ACRNN, a novel deep learning model for audio sentiment analysis that leverages parallel Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks to extract and fuse heterogeneous audio features. The model achieves state-of-the-art performance by creating a comprehensive "Audio Sentiment Vector" (ASV) that captures richer sentiment information than traditional methods.

What Problem Does This Paper Address?

Traditional sentiment analysis primarily relies on text. However, spoken language carries significant emotional cues beyond words, making audio sentiment analysis a crucial but challenging area. Existing approaches often focus on single types of acoustic features or fail to effectively fuse diverse feature sets. This limits their ability to capture the nuanced emotional content present in speech, especially when considering the contextual dependencies between utterances. The paper aims to bridge this gap by developing a model that can effectively extract and combine various acoustic features to achieve more accurate and robust audio sentiment classification.

Methods

The core of the proposed method is the AFF-ACRNN (Audio Feature Fusion-Attention based CNN and RNN) model, which employs a parallel architecture with two main branches:

  • Utterance-Based BiLSTM Branch (UB-BiLSTM): This branch utilizes Bidirectional Long Short-Term Memory (BiLSTM) networks to process sequences of acoustic features extracted from utterances. It captures temporal dependencies within and between utterances, aiming to learn sentiment information that evolves over time.
  • Spectrum-Based CNN Branch (SBCNN): This branch uses Convolutional Neural Networks (CNNs), specifically a ResNet152 architecture, to extract features from spectrum graphs (visual representations of audio signals). CNNs are adept at identifying spatial patterns, which in this context, translate to identifying discriminative patterns in the audio spectrum.

Crucially, the model incorporates an attention mechanism within the BiLSTM layers for feature fusion. This allows the model to dynamically weigh the importance of different features and contextual information, focusing on the most sentiment-relevant aspects. The outputs from both branches are fused to create a comprehensive Audio Sentiment Vector (ASV), which is then used for the final sentiment classification.

Datasets

The paper uses the following datasets for training and evaluation:

  • CMU-MOSI: A widely used dataset for multimodal sentiment analysis, containing opinionated utterances from videos. The authors use it for training and validation.
  • MOUD: A Spanish dataset of product review videos, used for testing and evaluating the model's generalization capabilities across languages.
  • Chinese Datasets (Sichuan, Cantonese, Mandarin): Three newly compiled Chinese datasets from social media platforms, used for expansion experiments to assess the model's performance on different languages and cultural contexts.

Key Insights and Ideas

  • Heterogeneous Feature Fusion is Key: The paper strongly advocates for combining diverse acoustic features (e.g., MFCC, spectral centroid, chroma features) rather than relying on a single type. This heterogeneity is crucial for capturing the multifaceted nature of emotional expression in speech.
  • Context Matters: The model's ability to leverage sequential information from BiLSTM and the attention mechanism highlights the importance of considering the context of an utterance within a larger audio segment or video. A single utterance's sentiment can be better understood when viewed alongside preceding and succeeding ones.
  • Spectrum Graphs Offer Complementary Information: The CNN branch's ability to extract features from spectrum graphs complements traditional acoustic features, suggesting that visual patterns in audio representations hold valuable sentiment-related information that might be missed by spectral analysis alone.
  • Attention as a Regulator: The attention mechanism acts as a sophisticated regulator, helping the model to focus on the most impactful features and contextual elements, thereby improving classification accuracy and potentially offering insights into why a particular sentiment is predicted.

Key Results

  • The proposed AFF-ACRNN model significantly outperforms traditional methods (SVM, Naive Bayes, GMM, HMM, DTW) on the MOUD dataset, achieving a 11.51% improvement in weighted accuracy over the best traditional model (SVM).
  • On the CMU-MOSI dataset, the AFF-ACRNN model achieves state-of-the-art performance, outperforming the previous best approach by 9.33% in binary classification.
  • The model demonstrates strong generalization capabilities, performing well on both English (MOSI) and Spanish (MOUD) datasets, and showing promising results on newly compiled Chinese datasets.
  • Extensive experiments on feature combinations reveal that a combination of four specific acoustic features (MFCC, spectral_centroid, spectral_contrast, and chroma_stft) yields the best results when used with the BiLSTM branch.

Limitations and Future Work

  • Language Dependence: While the model shows good generalization, the paper acknowledges the potential impact of different languages on generalization capabilities and conducts expansion experiments to address this.
  • Fusion Dimension: The authors suggest future work could explore fusing features across different dimensions and categories more deeply.
  • Multimodal Sentiment Analysis: The paper explicitly mentions applying the developed techniques to multimodal sentiment analysis, integrating audio with other modalities like text and video.

Summary updated

Abstract

Audio Sentiment Analysis is a popular research area which extends the conventional text-based sentiment analysis to depend on the effectiveness of acoustic features extracted from speech. However, current progress on audio sentiment analysis mainly focuses on extracting homogeneous acoustic features or doesn't fuse heterogeneous features effectively. In this paper, we propose an utterance-based deep neural network model, which has a parallel combination of Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) based network, to obtain representative features termed Audio Sentiment Vector (ASV), that can maximally reflect sentiment information in an audio. Specifically, our model is trained by utterance-level labels and ASV can be extracted and fused creatively from two branches. In the CNN model branch, spectrum graphs produced by signals are fed as inputs while in the LSTM model branch, inputs include spectral features and cepstrum coefficient extracted from dependent utterances in audio. Besides, Bidirectional Long Short-Term Memory (BiLSTM) with attention mechanism is used for feature fusion. Extensive experiments have been conducted to show our model can recognize audio sentiment precisely and quickly, and demonstrate our ASV is better than traditional acoustic features or vectors extracted from other deep learning models. Furthermore, experimental results indicate that the proposed model outperforms the state-of-the-art approach by 9.33\% on Multimodal Opinion-level Sentiment Intensity dataset (MOSI) dataset.
BibTeX
@misc{luo2018utterance,
  author = {Ziqian Luo and Hua Xu and Feiyang Chen},
  title = {{Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM}},
  year = {2018},
  eprint = {1811.08065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.08065},
}

Loading related content...

Figure