Visual Prompting in LLMs for Enhancing Emotion Recognition

Qixuan Zhang, Zhifeng Wang, Zhenyue Qin, Yang Liu, Tom Gedeon, Sabrina Caldwell

TL;DR

This paper introduces a novel "Set-of-Vision" (SoV) prompting technique for Vision-Language Models (VLLMs) to improve zero-shot emotion recognition. By combining bounding boxes, numbered identifiers, and facial landmarks, SoV guides VLLMs to precisely locate and analyze faces within their global context, leading to more accurate emotion detection and categorization compared to traditional methods.

What Problem Does This Paper Address?

Vision-Language Models (VLLMs) are powerful tools for understanding images and text, but accurately recognizing emotions within images remains a challenge. Existing methods often struggle with spatial localization of faces and can lose valuable global context by focusing too narrowly on individual faces. This can lead to misclassifications, especially in complex group settings or when faces are partially occluded. The authors aim to address this by developing a visual prompting strategy that enhances VLLMs' ability to interpret nuanced facial expressions while preserving the overall scene context.

Methods

The core contribution is the Set-of-Vision (SoV) prompting approach. This method enhances VLLMs for emotion recognition through a multi-stage process:

  1. Face Detection: Standard face detection algorithms (like RetinaFace) are used to identify faces in an image.
  2. Box Overlap Handling: A novel algorithm is proposed to manage overlapping bounding boxes from face detection. It prioritizes larger faces and discards smaller, potentially occluded ones, ensuring accurate face identification.
  3. Face Numbering: Each detected and validated face is assigned a unique numerical identifier. This helps differentiate individuals in group settings.
  4. Facial Landmark Detection: Key facial landmarks (e.g., eyes, nose, mouth) are detected for each face.
  5. Spatial Relationship Analysis: The positions of these facial landmarks are analyzed to understand spatial relationships, which are crucial for fine-grained emotion interpretation.

SoV then integrates these visual cues (numbered bounding boxes and facial landmarks) as prompts alongside text prompts for the VLLM. This allows the model to precisely ground its analysis on specific individuals and their facial features while still considering the broader image context.

Datasets

The authors collected original images from ABC News websites. These images were preprocessed to remove duplicates and blurry content. Emotion annotation was performed using DeepFace, followed by manual refinement by two human annotators and a final review by a third annotator with a psychology background. The dataset was structured into three difficulty levels: "easy" (3 or fewer faces), "medium" (3-7 faces), and "hard" (more than 7 faces).

Key Insights and Ideas

  • Spatial Context is Crucial: Simply identifying faces isn't enough; understanding their spatial relationships and the overall scene context significantly boosts emotion recognition accuracy.
  • Numbered Prompts Aid Differentiation: In group settings, assigning unique numbers to faces is vital for VLLMs to distinguish individuals and avoid confusion.
  • Facial Landmarks Provide Nuance: Beyond just bounding boxes, the precise locations of facial features offer critical cues for interpreting subtle emotional expressions.
  • Zero-Shot Performance Boost: SoV significantly enhances the zero-shot capabilities of VLLMs for emotion recognition, meaning it works well without task-specific fine-tuning.
  • Preserving Global Context: Unlike methods that crop faces, SoV operates on the entire image, allowing VLLMs to leverage background information for more informed emotion analysis.

Key Results

The SoV approach demonstrated significant improvements across various VLLMs, including GPT-4V.

  • Accuracy Gains: GPT-4V with SoV achieved an impressive 60.91% accuracy and 41.96% Recall@1 on the "easy" dataset, and maintained 50.00% accuracy and 18.97% Recall@1 on the "hard" dataset. This is a substantial improvement over the baseline GPT-4V (44.44% accuracy, 22.11% Recall@1).
  • Outperforming Other Prompts: SoV consistently outperformed other visual prompting methods like ReCLIP and RedCircle, especially in recognizing "Happy" and "Neutral" emotions.
  • Handling Complex Scenes: The SoV method proved effective in analyzing emotions in challenging scenarios, such as group settings with overlapping faces or partially visible faces.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Describing Visual Prompts: Precisely describing visual prompts (like shapes, colors, or locations) in language can be challenging, potentially requiring model fine-tuning for optimal performance.
  • Computational Intensity: The SoV method, with its multiple steps of detection and analysis, can be computationally intensive, which might limit its scalability for real-time applications or very large datasets.

Future work could explore more efficient encoding of visual prompts and investigate the application of SoV to other vision-language tasks beyond emotion recognition.

Summary updated

Abstract

Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing. Nonetheless, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. To address this problem, we propose a Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through a battery of experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model's ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.
BibTeX
@inproceedings{zhang2024visual,
  author = {Qixuan Zhang and Zhifeng Wang and Zhenyue Qin and Yang Liu and Tom Gedeon and Sabrina Caldwell},
  title = {{Visual Prompting in LLMs for Enhancing Emotion Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.257},
}
Figure