Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML)

Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, Stéphane Dupont, Edison Marrese-Taylor, Cristian Rodríguez-Opazo, Jorge Balazs, Stephen Jay Gould, Yutaka Matsuo, Eda Okur, Shachi H Kumar, Saurav Sahay, Lama Nachman, Anil George, Hemant Kumar, Kameshwar Rao, K Sowmya, S. V. L. Sunitha, Ashish Sardana, Stephen, Halbe, Shaunak, Yutaka, Rodriguez, Roesler, Oliver, Saurav, Shenoy, Aman, Singh, No, Tadas Baltrusaitis, Amir Zadeh, Yao, Chong Lim, Louis-Philippe Morency, Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Weinberger, Minghai Chen, Sen Wang, Paul Liang, Tadas Baltruaitis, P Ekman, Edmund Rolls, David Perrett, H Ellis, Deepanway Ghosal, Shad Md, Dushyant Akhtar, Soujanya Chauhan, Pushpak Poria, Asif Bhattacharyya, Ekbal, Diederik Kingma, Jimmy Ba, Rosalind Picard, Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Iti Chaturvedi, Amir Hussain, Sabrina Stckli, Michael Schulte-Mecklenbeck, Stefan Borer, Andrea Samson, Prateek Vij, Rowan Zellers, Eli Pincus, Carlos BussoOpenalex Unknown 2020 · DOI · OpenAlex

Abstract

Understanding expressed sentiment and emotions are two crucial factors in human multimodal language. This paper describes a Transformer-based joint-encoding (TBJE) architecture designed for the tasks of Emotion Recognition and Sentiment Analysis. By utilizing modular co-attention and a specialized glimpse layer to jointly encode multiple modalities, the proposed model offers a computationally efficient alternative to traditional recurrent architectures. Evaluated on the CMU-MOSEI dataset, the TBJE model demonstrates state-of-the-art performance, particularly when leveraging linguistic features to anchor acoustic signals.

Introduction

Human communication is a rich, multi-layered process where meaning is derived not only from spoken words but also from vocal inflection and facial expressions. In the field of Natural Language Processing (NLP), "Multimodal Language" research seeks to model these complex interactions to better understand human affect. Traditionally, this field has been dominated by Recurrent Neural Networks (RNNs), such as LSTMs. However, RNNs face significant limitations: they are difficult to parallelize and often struggle to capture the intricate cross-modal dynamics—such as how a specific word's meaning might be inverted by a sarcastic tone.

To address these challenges, we introduce the Transformer-Based Joint-Encoding (TBJE) model. By replacing sequential processing with a Transformer-based architecture, we achieve a model that is faster to train and more adept at jointly encoding disparate data streams.

The TBJE Architecture

The TBJE model is built upon three primary structural components designed to process and fuse multimodal information efficiently.

1. Monomodal Transformers

The first stage involves standard Transformer blocks consisting of self-attention mechanisms and Multi-Layer Perceptrons (MLPs). These are used to encode individual modalities—Linguistic ($L$), Acoustic ($A$), and Visual ($V$)—independently before fusion.

2. Modular Co-Attention

A core innovation of the TBJE is the use of the linguistic modality as an "anchor." Instead of processing modalities in isolation, the model employs modular co-attention where the linguistic features ($x$) modulate the other modalities ($y$). By switching the "Key" and "Context" in the attention mechanism, the representation of audio or video is conditioned directly on the text being spoken. This ensures that the model interprets non-verbal cues within the specific context of the words used.

3. Glimpse Layers

Following the modality blocks, the architecture incorporates a "glimpse layer." This layer utilizes multiple soft-attention mechanisms to project the modality into a refined representation space. This effectively "summarizes" the most salient features of the signal before they are passed to the final classification head.

Experimental Setup and Dataset

The model was evaluated using the CMU-MOSEI (Multimodal Opinion Sentiment and Emotion Intensity) dataset, the largest benchmark of its kind.

  • Dataset Scale: It contains 23,453 annotated sentences from 1,000 distinct speakers.
  • Labels: Sentiment is rated on a 7-point scale (from -3 to +3), while emotions are categorized into six classes: happiness, sadness, anger, fear, disgust, and surprise.
  • Feature Extraction: The study utilized GloVe embeddings for text, mel-spectrograms for audio, and a pre-trained R(2+1)D-152 CNN for visual features.

Results and Discussion

The experiments revealed that the L+A (Linguistic + Acoustic) configuration was the most effective variant of the TBJE architecture.

Performance Metrics

In sentiment analysis, the TBJE (L+A) achieved 82.4% accuracy on binary sentiment classification and 45.5% on 7-class sentiment, outperforming previous benchmarks like the Graph-MFN model. In emotion recognition, the model showed particularly high accuracy in detecting "Fear" (89.2%) and "Happy" (66.0%).

The "Visual Integration" Wall

An unexpected finding was that the inclusion of visual data ($L+A+V$) actually led to a decrease in performance compared to the $L+A$ model. This suggests that visual features, such as body movement or facial landmarks, currently introduce significant noise. Integrating these features effectively remains a primary challenge in multimodal research, requiring better extraction or more sophisticated fusion techniques.

Efficiency and Scaling

By moving away from recurrence, the TBJE model is highly parallelizable and computationally efficient. Ablation studies indicated that 6 Transformer blocks provided the optimal balance between depth and accuracy; adding more blocks led to diminishing returns or slight performance drops.

Conclusion and Future Work

The TBJE model demonstrates that Transformers can effectively anchor multimodal signals using text, providing a scalable and efficient solution for affective computing. However, limitations remain. The current model processes utterances in isolation, lacking the conversational context that models like Multilogue-Net capture. Future work will focus on incorporating temporal context across dialogue turns and refining the visual pipeline to better filter irrelevant information.

Summary updated

Abstract

Understanding expressed sentiment and emotions are two crucial factors in human multimodal language.This paper describes a Transformer-based joint-encoding (TBJE) for the task of Emotion Recognition and Sentiment Analysis.In addition to use the Transformer architecture, our approach relies on a modular co-attention and a glimpse layer to jointly encode one or more modalities.The proposed solution has also been submitted to the ACL20: Second Grand-Challenge on Multimodal Language to be evaluated on the CMU-MOSEI dataset.The code to replicate the presented experiments is open-source 1 .
BibTeX
@inproceedings{delbrouck2020second,
  author = {Jean-Benoit Delbrouck and Noé Tits and Mathilde Brousmiche and Stéphane Dupont and Edison Marrese-Taylor and Cristian Rodríguez-Opazo and Jorge Balazs and Stephen Jay Gould and Yutaka Matsuo and Eda Okur and Shachi H Kumar and Saurav Sahay and Lama Nachman and Anil George and Hemant Kumar and Kameshwar Rao and K Sowmya and S. V. L. Sunitha and Ashish Sardana and Gould and Stephen and H Kumar and Halbe and Shaunak and Kameshwar Rao and Kamath Sowmya and Kumar and Matsuo and Yutaka and Selvan Sunitha and Rodriguez and Roesler and Oliver and Sahay and Saurav and Ashish Sardana and Shenoy and Aman and Singh and Tits and No and Tadas Baltrusaitis and Amir Zadeh and Yao and Chong Lim and Louis-Philippe Morency and Gao Huang and Zhuang Liu and Laurens Van Der Maaten and Kilian Weinberger and Minghai Chen and Sen Wang and Paul Liang and Tadas Baltruaitis and Amir Zadeh and Louis-Philippe Morency and P Ekman and Edmund Rolls and David Perrett and H Ellis and Deepanway Ghosal and Shad Md and Dushyant Akhtar and Soujanya Chauhan and Pushpak Poria and Asif Bhattacharyya and Ekbal and Diederik Kingma and Jimmy Ba and Rosalind Picard and Soujanya Poria and Erik Cambria and Devamanyu Hazarika and Navonil Majumder and Amir Zadeh and Louis-Philippe Morency and Soujanya Poria and Iti Chaturvedi and Erik Cambria and Amir Hussain and Sabrina Stckli and Michael Schulte-Mecklenbeck and Stefan Borer and Andrea Samson and Amir Zadeh and Minghai Chen and Soujanya Poria and Erik Cambria and Louis-Philippe Morency and Amir Zadeh and Paul Liang and Soujanya Poria and Prateek Vij and Erik Cambria and Louis-Philippe Morency and Amir Zadeh and Rowan Zellers and Eli Pincus and Louis-Philippe Morency and Carlos Busso},
  title = {{Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML)}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.18653/v1/2020.challengehml-1},
}
Figure