Learning by Comparing: Boosting Multimodal Affective Computing through Ordinal Learning

Sijie Mai, Ying Zeng, Haifeng Hu

TL;DR

This paper introduces a novel Multimodal Ordinal Affective Computing (MOAC) framework that leverages ordinal learning to better understand human affective states. By teaching models to "learn by comparing" rather than just predicting absolute labels, MOAC significantly improves performance on sentiment analysis, humor detection, and sarcasm detection tasks.

What Problem Does This Paper Address?

Multimodal Affective Computing (MAC) aims to understand human emotions and opinions by integrating information from various sources like text, audio, and video. However, existing methods often treat affective states as discrete or continuous values, neglecting their inherently subjective and ordinal nature. For instance, humans often perceive emotions on a relative scale (e.g., "happy" is more intense than "content") rather than assigning an absolute score. This simplification can lead to models that struggle with the nuances of human affect. This paper addresses this gap by proposing a framework that explicitly models the ordinal relationships between affective states.

Methods

The core of the MOAC framework lies in its two-stage ordinal learning approach:

  • Label-level Ordinal Learning: This stage focuses on learning to compare samples in the label space. Instead of predicting a precise score, the model is trained to assign higher predicted values to samples with higher annotated labels. This encourages the model to grasp the relative intensity of affective states. A regularization loss is also introduced to keep the model's output distributions aligned with the ground truth.
  • Feature-level Ordinal Learning: This stage refines the learning at the feature level. It employs two key techniques:
  • Feature Difference Operation: This involves calculating the difference between the multimodal representations of two samples. This "new" feature is then used to predict the label difference, leading to more robust training and better feature representation.
  • Neutral Embedding: A novel neutral embedding is introduced to simplify prediction. It helps the model estimate the difference between a target representation and a neutral reference, making it easier to infer the polarity and intensity of affective states.

Datasets

The MOAC framework was evaluated on several established multimodal datasets:

  • CMU-MOSI [65]: A widely used dataset for Multimodal Sentiment Analysis (MSA), containing over 2,000 video utterances with sentiment scores ranging from -3 to 3.
  • CMU-MOSEI [64]: A larger MSA dataset with over 22,000 video utterances from over 1,000 speakers, annotated for both emotions and sentiment scores (-3 to 3).
  • UR-FUNNY [14]: Used for Multimodal Humor Detection (MHD), derived from TED talk videos, with "laughter" tags indicating humorous punchlines.
  • MUSTARD [5]: Used for Multimodal Sarcasm Detection (MSD), featuring video utterances from TV series labeled as sarcastic or non-sarcastic.

Key Insights and Ideas

  • Ordinal Nature of Affect: The paper strongly advocates for the idea that human affective states are inherently ordinal, and models should be designed to reflect this. This is a fundamental shift from traditional regression or classification approaches.
  • "Learning by Comparing": The core innovation is teaching models to learn by comparing samples, mirroring human intuition. This is more effective than simply trying to map inputs to absolute labels, especially for subjective concepts like emotions.
  • Two-Stage Learning for Depth: The separation into label-level and feature-level ordinal learning provides a structured approach, first building a foundational understanding of relative affective states and then refining it with more nuanced feature representations.
  • Neutral Embedding for Simplicity: The introduction of a neutral embedding is a clever way to simplify the prediction task by providing a reference point, making it easier for the model to discern subtle differences in affective intensity.

Key Results

The MOAC framework demonstrated state-of-the-art performance across multiple tasks and datasets:

  • Multimodal Sentiment Analysis (MSA): MOAC consistently outperformed competitive baselines on CMU-MOSI and CMU-MOSEI datasets, showing significant improvements in accuracy (Acc7, Acc2) and F1 scores. For example, on CMU-MOSI, it outperformed ITHP by up to 2.5% in Acc2 and F1 score.
  • Multimodal Humor Detection (MHD) and Sarcasm Detection (MSD): MOAC achieved state-of-the-art results on the UR-FUNNY and MUSTARD datasets, outperforming methods like HKT, MCL, and MGCL.
  • Ablation Studies: Experiments confirmed the importance of each component of MOAC, with the removal of ordinal learning leading to significant performance drops. The feature-level ordinal learning, particularly the feature difference operation and neutral embedding, proved crucial.
  • Visualization: t-SNE visualizations showed that MOAC leads to more distinct separation of positive and negative sentiment clusters and tighter intra-cluster distributions compared to models without ordinal learning.

Limitations and Future Work

  • Acc7 Performance: While MOAC achieved strong results, the paper notes that its Acc7 scores on CMU-MOSI and CMU-MOSEI were not the absolute highest. This is hypothesized to be due to the intentional introduction of a margin in the ordinal learning, which allows for some error to facilitate learning but might not be optimal for fine-grained sentiment labels.
  • Neutral Utterance Ambiguity: The paper acknowledges that even with MOAC, there can be some overlap and ambiguity with neutral data points, especially when they are not clearly differentiated from weakly positive or negative samples. This is attributed to the limited number of neutral utterances in datasets.
  • Future Work: The authors suggest further research into developing more effective representations for neutral utterances, especially in scenarios with insufficient data. They also highlight the potential for extending MOAC to other affective computing tasks.

Summary updated

Abstract

Multimodal affective computing aims to integrate information from multiple modalities for the analysis of human affective states, opinion tendencies, behavior intentions, etc. Previous studies primarily focus on approximating predictions to annotated labels, often neglecting the ordinal nature of affective states. In this paper, we address this issue by exploring ordinal learning, and a Multimodal Ordinal Affective Computing (MOAC) framework is designed to enhance the understanding of the nature of affective concepts. Specifically, we propose coarse-grained label-level ordinal learning that prompts the model to \textit{learn to compare} in the label space, encouraging higher predictive values for samples annotated with larger labels over those with smaller labels. Moreover, a regularization loss is proposed to prevent the output distributions from deviating significantly from the annotated label distributions. Fine-grained feature-level ordinal learning is then performed via the feature difference operation and the neutral embedding. The former compares samples in the feature space, calculating the difference between features of different samples to generate `new' features for a more robust training. The latter seeks to reduce the difficulty of prediction by estimating the difference between the target multimodal representations and a neutral reference. We first demonstrate MOAC in multimodal sentiment analysis, which is a regression task that aligns well with the function of ordinal learning. Then we extend MOAC to classification tasks including multimodal humor detection and sarcasm detection to evaluate its generalizability. Experiments suggest that MOAC outperforms state-of-the-art methods.
BibTeX
@inproceedings{mai2025learning,
  author = {Sijie Mai and Ying Zeng and Haifeng Hu},
  title = {{Learning by Comparing: Boosting Multimodal Affective Computing through Ordinal Learning}},
  booktitle = {The Web Conference},
  year = {2025},
  doi = {10.1145/3696410.3714841},
}

Loading related content...

Figure