Face-Cap: Image Captioning Using Facial Expression Analysis

Omid Mohamad Nezami, Mark Dras, Peter Anderson, Len Hamey

TL;DR

This paper introduces "Face-Cap," a novel image captioning system that leverages facial expressions to generate more human-like descriptions. By incorporating facial emotion analysis, Face-Cap models significantly improve caption quality, not by adding emotional adjectives, but by generating more diverse descriptions of actions.

What Problem Does This Paper Address?

Traditional image captioning systems excel at describing the factual content of an image, such as objects and their spatial relationships. However, they often overlook the emotional context, which is crucial for understanding human activities and interpersonal interactions. This emotional dimension can make captions richer and more engaging. Existing approaches that try to incorporate sentiment typically rely on external sentiment lexicons or require specialized datasets, which are difficult to obtain. This paper aims to bridge this gap by directly using facial expressions within an image to inform the captioning process, without relying on external sentiment resources.

Methods

The core of the Face-Cap system involves two main components: a facial expression recognition (FER) model and an image captioning model.

  1. Facial Expression Recognition (FER): The authors train a VGGnet-based model on the FER-2013 dataset, which contains "in-the-wild" images with labeled emotions. This model achieves a respectable 72.7% accuracy, outperforming human performance on the same dataset. This trained FER model is then used as a feature extractor to obtain probabilities for seven universal emotions (happiness, sadness, fear, surprise, anger, disgust, neutral) from faces detected in the images. These probabilities are then converted into a one-hot encoded vector representing the aggregate facial emotion of the image.
  2. Image Captioning Model (Face-Cap): The captioning model is based on a Long Short-Term Memory (LSTM) network, similar to state-of-the-art attention-based models. The key innovation is how facial expression features are integrated:
  3. Face-CapF: This variant feeds the facial expression features into the LSTM at every time step, alongside image features and previous word embeddings. It also incorporates a "face loss function" during training to encourage the model to learn from these facial cues.
  4. Face-CapL: This variant is similar but removes the facial features from being fed at every time step. Instead, it uses them to initialize the LSTM's memory and hidden states, allowing the model to "forget" or ignore irrelevant facial information during generation.

The image features themselves are extracted using an Oxford VGGnet pre-trained on ImageNet and are weighted using an attention mechanism, allowing the model to focus on relevant image regions for each word it generates.

Datasets

  • FER-2013: Used for training the facial expression recognition model. It contains 35,887 images with labels for six basic emotions plus neutral.
  • FlickrFace11K: A newly created dataset for training and evaluating the image captioning models. It's a subset of the Flickr 30K dataset, specifically filtered to include images containing human faces. It comprises approximately 11,000 image-caption pairs. This dataset is made publicly available.

Key Insights and Ideas

  • Emotional Context Enhances Action Descriptions: Surprisingly, the improved caption quality from Face-Cap models didn't stem from adding emotional adjectives (e.g., "happy," "sad"). Instead, the models generated captions with a greater variety of described actions. This suggests that understanding the emotional state of individuals in an image can lead to a richer understanding of what they are doing.
  • Facial Features as a Rich Source of Information: Directly using facial expressions, rather than relying on external sentiment lexicons, proves effective. This approach captures a more nuanced and direct emotional signal from the image content.
  • Selective Use of Emotional Cues is Beneficial: The Face-CapL model, which can selectively ignore facial features when irrelevant, performs slightly better than Face-CapF, which forces the model to consider facial features at every step. This highlights the importance of allowing the model flexibility in how it utilizes emotional information.
  • Novel Dataset for Emotion-Aware Captioning: The creation and release of FlickrFace11K provide a valuable resource for future research in this specific area.

Key Results

  • Outperformance of Baseline: Both Face-CapF and Face-CapL models consistently outperformed the state-of-the-art baseline (Xu's model) across standard evaluation metrics like BLEU, ROUGE-L, METEOR, CIDEr, and SPICE.
  • Face-CapL Leads: Face-CapL generally achieved the best results, particularly in metrics like CIDEr, which penalizes generic captions.
  • Action Diversity: Linguistic analysis revealed that Face-Cap models generated captions with a higher entropy in verb distributions, indicating more varied action descriptions. Verbs like "smiling," "singing," and "looking" were more frequently and appropriately generated by the Face-Cap models.

Limitations and Future Work

  • Limited Emotion Set: The current models are trained on a fixed set of seven basic emotions. Future work could explore a richer set of emotions, including confusion and curiosity.
  • Potential for Misinterpretation: The paper acknowledges instances where facial features might be misinterpreted due to visual likeness (e.g., mistaking a visual cue for "holding a microphone" when it's not).
  • Soft Injection of Features: The authors suggest exploring alternative architectures for injecting facial emotions, such as a "soft injection" approach, which might offer more nuanced integration.
  • Improving Emotion Recognition: Developing more robust facial expression recognition models for diverse "in-the-wild" scenarios could further enhance captioning performance.

Summary updated

Abstract

Image captioning is the process of generating a natural language description of an image. Most current image captioning models, however, do not take into account the emotional aspect of an image, which is very relevant to activities and interpersonal relationships represented therein. Towards developing a model that can produce human-like captions incorporating these, we use facial expression features extracted from images including human faces, with the aim of improving the descriptive ability of the model. In this work, we present two variants of our Face-Cap model, which embed facial expression features in different ways, to generate image captions. Using all standard evaluation metrics, our Face-Cap models outperform a state-of-the-art baseline model for generating image captions when applied to an image caption dataset extracted from the standard Flickr 30K dataset, consisting of around 11K images containing faces. An analysis of the captions finds that, perhaps surprisingly, the improvement in caption quality appears to come not from the addition of adjectives linked to emotional aspects of the images, but from more variety in the actions described in the captions.
BibTeX
@inproceedings{nezami2018face,
  author = {Omid Mohamad Nezami and Mark Dras and Peter Anderson and Len Hamey},
  title = {{Face-Cap: Image Captioning Using Facial Expression Analysis}},
  booktitle = {ECML/PKDD},
  year = {2018},
  doi = {10.1007/978-3-030-10925-7\_14},
}

Loading related content...

Figure