Towards Deeper Emotional Reflection: Crafting Affective Image Filters with Generative Priors

Peixuan Zhang, Shuchen Weng, Jiajun Tang, Si Li, Boxin Shi

Abstract

The integration of text and imagery is a cornerstone of digital expression, yet existing tools often fail to capture the nuanced emotional resonance of descriptive language. This paper introduces the Affective Image Filter (AIF) task, which seeks to transform images based on visually-abstract emotional cues found in text. We propose AIF-D, a sophisticated diffusion-based model that evolves from a transformer-based baseline (AIF-B). By leveraging generative priors and novel architectural modules, AIF-D effectively "re-colors" and "re-textures" photographs to match complex human feelings while maintaining structural integrity. Our results demonstrate significant improvements in structural consistency, emotional fidelity, and user preference over existing state-of-the-art methods.

Introduction

While social media platforms offer a variety of standard image filters, these are typically limited to generic aesthetic adjustments like "sepia" or "high contrast." There is a notable lack of tools capable of translating nuanced, emotionally-charged sentences—such as "The faded photographs on the crumbling walls reflected the melancholy of a bygone era"—into precise visual transformations.

The researchers identify three primary challenges in affective image editing that current models struggle to overcome:

  1. Blurry Details: Downsampling during processing often leads to the loss of high-frequency textures and fine details.
  2. Keyword Blindness: Models frequently fixate on isolated words (e.g., "warm") rather than interpreting the broader emotional context of the entire sentence.
  3. Overly-stylized Effects: Previous attempts often resulted in "deep-fried" images characterized by exaggerated contrast and distorted content.
Figure 1: Comparison of AIF-B and AIF-D across common affective editing challenges: (a) Blurry details, (b) Keyword blindness, and (c) Overly-stylized effects.

Methodology

The paper explores two iterations of affective filtering models:

AIF-B: The Baseline

AIF-B utilizes a multi-modal transformer architecture. It maps words to emotional scores using the VAD dictionary (Valence, Arousal, Dominance) and employs a sentiment metric loss to align the output image's "feel" with the input text.

AIF-D: The Advanced Diffusion Model

To address the limitations of AIF-B, the authors developed AIF-D using a Latent Diffusion Model (LDM) backbone. Several key innovations distinguish this model:

  • Content Preservation Module (CPM): This module injects multi-scale spatial features from the original image directly into the U-Net, preventing the loss of structural detail during emotional transformation.
  • In-depth Emotional Reasoning: Instead of simple keyword matching, AIF-D uses a Large Language Model (LLM) with Chain-of-Thought (CoT) prompting to expand user text into rich, descriptive emotional contexts.
  • Voting Ensemble Mechanism: To account for the subjectivity of emotion, the model aggregates "votes" from various perspectives—including color histograms, texture matrices, and style features—to determine the target emotional distribution.
  • Emotional Reflection Decoder: This fine-tuning strategy balances artistic expression with content consistency by applying a "texture mapping loss" across multiple stages of the image decoder.

Datasets and Evaluation

The research utilizes a newly curated AIF Dataset derived from ArtEmis and ArtEmis v2, containing 69.6K images and 292.9K text descriptions. These were manually filtered to ensure high-quality emotional labels based on Mikel’s wheel, a standard psychological model of eight emotional categories. For real-world testing, the COCO-Stuff dataset provided content images to evaluate the filters' applicability to general photography.

Key Insights

A central finding is that generative priors are essential. Rather than training a model from scratch to understand "sadness," it is more effective to steer a model already rich in generative knowledge, like Stable Diffusion. Furthermore, the Voting Ensemble highlights that emotion is multi-faceted; for instance, "fear" may rely more on jagged textures, whereas "peace" is better represented through smooth color gradients. Finally, the authors found a "sweet spot" for emotional expression by applying artistic losses at specific stages of the decoding process to avoid content distortion.

Results

AIF-D outperformed several state-of-the-art image editing methods, including ManiGAN and DiffusionCLIP:

  • Structural Consistency: AIF-D achieved an SSIM of 57.74%, significantly higher than AIF-B (56.15%) and ManiGAN (50.72%).
  • Emotional Fidelity: In Ensemble Accuracy (EAcc) tests, AIF-D correctly reflected the intended emotion 37.48% of the time.
  • User Preference: A user study showed an Emotional Preference Score (EPS) of 39.51% for AIF-D, nearly doubling the score of the next best model.

Conclusion and Future Work

AIF-D represents a significant step toward more expressive and context-aware image editing. While the subjectivity of emotion and the computational intensity of diffusion models remain challenges, future work aims to expand these filters to video and explore deeper semantic changes, such as altering objects within a scene to better match an emotional narrative.

Summary updated

BibTeX
@article{zhang2025towards,
  author = {Peixuan Zhang and Shuchen Weng and Jiajun Tang and Si Li and Boxin Shi},
  title = {{Towards Deeper Emotional Reflection: Crafting Affective Image Filters with Generative Priors}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  doi = {10.1109/tpami.2025.3643911},
}
Figure