Multiple Feature Refining Network for Visual Emotion Distribution Learning

Qinfu Xu, Shaozu Yuan, Yiwei Wei, Jie Wu, Leiquan Wang, Chunlei WuAAAI 2025 · DOI

TL;DR

This paper introduces the Multiple Feature Refining Network (MFRN), a novel approach to visual emotion distribution learning (VEDL) that tackles the issue of information redundancy and limited contextual understanding in existing methods. MFRN employs a "Spectral Mixer" to disentangle low- and high-level features and a "Semantic Graph Prompt Learning" module to capture object-attribute relationships, achieving state-of-the-art results on multiple VEDL benchmarks.

What Problem Does This Paper Address?

Visual emotion distribution learning (VEDL) aims to understand the complex and nuanced emotions conveyed by images, recognizing that a single image can evoke a range of sentiments. Existing VEDL methods, primarily based on Convolutional Neural Networks (CNNs) or attention mechanisms, often struggle with two key issues:

  1. Information Redundancy: Deep networks tend to learn overlapping features across different layers, leading to noisy representations that obscure subtle emotional cues. This is exacerbated by the fact that even attention mechanisms can inadvertently focus on irrelevant details.
  2. Limited Contextual Understanding: While current models can capture local and global image features, effectively integrating them to understand the dynamic interplay of various emotions within an image remains a challenge. This leads to an incomplete understanding of fine-grained emotional semantics.

The paper argues that existing methods lack effective "information filtering" from an emotional perspective, hindering the ability to learn rich and accurate emotional distributions.

Methods

The proposed Multiple Feature Refining Network (MFRN) addresses these limitations through two core modules:

  1. Spectral Mixer: This module leverages wavelet transformation (specifically, Dual-Tree Complex Wavelet Transform - DTCWT) to decompose image features into low-frequency (LF) and high-frequency (HF) components.
  2. LF features capture global information like brightness and contrast.
  3. HF features capture fine-grained details like textures and patterns.
  4. This separation helps filter out redundant low-level information and enhances emotion saliency.
  5. A self-attention mechanism is then applied to align features across frequencies, followed by a channel-wise MLP for feature fusion.
  6. Semantic Graph Prompt Learning: This module focuses on high-level semantic understanding by constructing a graph representation of the image.
  7. Object Detection: An object detection toolkit (Faster-RCNN) is used to extract object and attribute features, which serve as nodes in the graph.
  8. Graph Construction: A visual graph is built where nodes represent objects and attributes. Edges are defined based on cosine similarity of attributes and IoU scores of object regions, capturing relationships between visual elements.
  9. Graph Attention Networks (GAT): A multi-layer GAT is employed to learn from these graph structures, dynamically adjusting the fusion of node information.
  10. Prompt Learning: To inject emotion-specific biases, a "graph prompt learning" method is introduced. This involves adding learnable prompt parameters to the graph nodes, allowing the model to adapt its understanding towards emotional semantics. The paper explores different prompt strategies, with a "Unified Prompt" showing the best performance.

The overall MFRN architecture combines the outputs of the Spectral Mixer and Semantic Graph Prompt Learning modules for final emotion distribution prediction.

Datasets

The paper evaluates MFRN on three publicly available datasets for visual emotion distribution learning:

  • Emotion6: Contains 1,980 images annotated with 7 emotion classes.
  • Flickr-LDL: Contains 11,150 images annotated with 7 emotion classes.
  • Twitter-LDL: Contains 10,045 images annotated with 7 emotion classes.

The authors note that the performance improvement on Twitter-LDL was less pronounced, potentially due to the more abstract and complex nature of its images.

Key Insights and Ideas

  • Frequency Domain for Emotion: Decomposing image features into frequency components (LF/HF) is a powerful way to filter out redundant low-level information and highlight emotionally relevant features, which is crucial for VEDL.
  • Graph Reasoning for Context: Building explicit graph structures that link objects and their attributes provides a robust way to capture contextual relationships, which are vital for understanding complex emotional scenes.
  • Prompting for Emotion Bias: Injecting learned "prompts" into the graph reasoning process is an effective way to guide the model towards learning emotion-specific semantics, moving beyond general object-attribute relationships.
  • Dual-Loss Strategy: Combining a distribution loss (KL divergence) with a classification loss (cross-entropy) for the dominant emotion helps the model learn both the overall distribution and the most prominent sentiment, leading to more accurate predictions.

Key Results

MFRN achieves state-of-the-art (SOTA) performance on all three tested datasets (Emotion6, Flickr-LDL, and Twitter-LDL) across various evaluation metrics, including Chebyshev distance, Clark distance, Canberra metric, KL divergence, cosine coefficient, and intersection similarity. The paper highlights that MFRN outperforms previous methods, including those based on CNNs and attention mechanisms, and even large multimodal models. Ablation studies confirm the effectiveness of both the Spectral Mixer and the Semantic Graph Prompt Learning modules, as well as the proposed dual-loss strategy.

Limitations and Future Work

  • Limitations:
    • The performance improvement on the Twitter-LDL dataset was less significant compared to others, possibly due to the abstract nature of its images and less explicit semantics.
  • While the paper explores different prompt learning strategies, the optimal prompt injection methodology for VEDL is still an area for further investigation.
  • Future Work:
    • The authors suggest improving visual emotion distribution learning by leveraging large multimodal models to obtain better data annotations.
  • Further exploration into prompt learning methodologies for graph modeling is also indicated.

Summary updated

BibTeX
@inproceedings{xu2025multiple,
  author = {Qinfu Xu and Shaozu Yuan and Yiwei Wei and Jie Wu and Leiquan Wang and Chunlei Wu},
  title = {{Multiple Feature Refining Network for Visual Emotion Distribution Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.32965},
}
Figure