Abstract
Understanding expressed sentiment and emotions are two crucial factors in human multimodal language. This paper describes a Transformer-based joint-encoding (TBJE) architecture designed for the tasks of Emotion Recognition and Sentiment Analysis. By utilizing modular co-attention and a specialized glimpse layer to jointly encode multiple modalities, the proposed model offers a computationally efficient alternative to traditional recurrent architectures. Evaluated on the CMU-MOSEI dataset, the TBJE model demonstrates state-of-the-art performance, particularly when leveraging linguistic features to anchor acoustic signals.
Introduction
Human communication is a rich, multi-layered process where meaning is derived not only from spoken words but also from vocal inflection and facial expressions. In the field of Natural Language Processing (NLP), "Multimodal Language" research seeks to model these complex interactions to better understand human affect. Traditionally, this field has been dominated by Recurrent Neural Networks (RNNs), such as LSTMs. However, RNNs face significant limitations: they are difficult to parallelize and often struggle to capture the intricate cross-modal dynamics—such as how a specific word's meaning might be inverted by a sarcastic tone.
To address these challenges, we introduce the Transformer-Based Joint-Encoding (TBJE) model. By replacing sequential processing with a Transformer-based architecture, we achieve a model that is faster to train and more adept at jointly encoding disparate data streams.
The TBJE Architecture
The TBJE model is built upon three primary structural components designed to process and fuse multimodal information efficiently.
1. Monomodal Transformers
The first stage involves standard Transformer blocks consisting of self-attention mechanisms and Multi-Layer Perceptrons (MLPs). These are used to encode individual modalities—Linguistic ($L$), Acoustic ($A$), and Visual ($V$)—independently before fusion.
2. Modular Co-Attention
A core innovation of the TBJE is the use of the linguistic modality as an "anchor." Instead of processing modalities in isolation, the model employs modular co-attention where the linguistic features ($x$) modulate the other modalities ($y$). By switching the "Key" and "Context" in the attention mechanism, the representation of audio or video is conditioned directly on the text being spoken. This ensures that the model interprets non-verbal cues within the specific context of the words used.
3. Glimpse Layers
Following the modality blocks, the architecture incorporates a "glimpse layer." This layer utilizes multiple soft-attention mechanisms to project the modality into a refined representation space. This effectively "summarizes" the most salient features of the signal before they are passed to the final classification head.
Experimental Setup and Dataset
The model was evaluated using the CMU-MOSEI (Multimodal Opinion Sentiment and Emotion Intensity) dataset, the largest benchmark of its kind.
- Dataset Scale: It contains 23,453 annotated sentences from 1,000 distinct speakers.
- Labels: Sentiment is rated on a 7-point scale (from -3 to +3), while emotions are categorized into six classes: happiness, sadness, anger, fear, disgust, and surprise.
- Feature Extraction: The study utilized GloVe embeddings for text, mel-spectrograms for audio, and a pre-trained R(2+1)D-152 CNN for visual features.
Results and Discussion
The experiments revealed that the L+A (Linguistic + Acoustic) configuration was the most effective variant of the TBJE architecture.
Performance Metrics
In sentiment analysis, the TBJE (L+A) achieved 82.4% accuracy on binary sentiment classification and 45.5% on 7-class sentiment, outperforming previous benchmarks like the Graph-MFN model. In emotion recognition, the model showed particularly high accuracy in detecting "Fear" (89.2%) and "Happy" (66.0%).
The "Visual Integration" Wall
An unexpected finding was that the inclusion of visual data ($L+A+V$) actually led to a decrease in performance compared to the $L+A$ model. This suggests that visual features, such as body movement or facial landmarks, currently introduce significant noise. Integrating these features effectively remains a primary challenge in multimodal research, requiring better extraction or more sophisticated fusion techniques.
Efficiency and Scaling
By moving away from recurrence, the TBJE model is highly parallelizable and computationally efficient. Ablation studies indicated that 6 Transformer blocks provided the optimal balance between depth and accuracy; adding more blocks led to diminishing returns or slight performance drops.
Conclusion and Future Work
The TBJE model demonstrates that Transformers can effectively anchor multimodal signals using text, providing a scalable and efficient solution for affective computing. However, limitations remain. The current model processes utterances in isolation, lacking the conversational context that models like Multilogue-Net capture. Future work will focus on incorporating temporal context across dialogue turns and refining the visual pipeline to better filter irrelevant information.
Summary updated