Reinforced Context Augmentation for Multimodal Emotion Analysis
Lists
Only you see this
Abstract
The task of multimodal emotion analysis aims to identify the emotions carried by the multimodal content, such as images and texts, which plays a crucial role in human-computer interaction and emotion-aware applications. While being important, emotion analysis is normally challenging, especially in circumstances where content from different modalities are insufficient and limited contextual information restricts the modal in accurately prediction. Existing approaches often rely on independent processing of unimodal features or fusing them, where rare additional information is leveraged to enhance the semantics of the multimodal content for this task. Therefore, it is expected to have effective content complementing solutions to deal with this situation for better understanding from multiple modalities. In this paper, we propose a data augmentation approach that enhances emotion analysis by providing additional contextual information. Specifically, we employ a context generator based on a large language model (LLM) to produce complementary text, which serves as enrichment of the input and is sent to the emotion classifier together with the original image and text. To ensure higher-quality context generation, we utilize reinforcement learning for the optimization of the generator with three types of reward, i.e., visual descriptivity that measures the alignment between augmented context and image, context complementarity that assesses the diversity between augmented context and the original text, and prediction ability that corresponds to the contribution of augmented context to emotion prediction. We evaluate our approach on two public multimodal emotion datasets, namely, TumEmo and MSED, where state-of-the-art performance is observed. Further analysis also demonstrates the validity and effectiveness of our approach.
BibTeX
@article{gan2026reinforced,
author = {Ruyi Gan and Yuanhe Tian and Kunhao Pan and Yan Song and Yong Zhang},
title = {{Reinforced Context Augmentation for Multimodal Emotion Analysis}},
journal = {IEEE Transactions on Multimedia},
year = {2026},
doi = {10.1109/tmm.2026.3673543},
}Loading related content...