TL;DR
This paper introduces "Face-Cap," a novel image captioning system that leverages facial expressions to generate more human-like descriptions. By incorporating facial emotion analysis, Face-Cap models significantly improve caption quality, not by adding emotional adjectives, but by generating more diverse descriptions of actions.
What Problem Does This Paper Address?
Traditional image captioning systems excel at describing the factual content of an image, such as objects and their spatial relationships. However, they often overlook the emotional context, which is crucial for understanding human activities and interpersonal interactions. This emotional dimension can make captions richer and more engaging. Existing approaches that try to incorporate sentiment typically rely on external sentiment lexicons or require specialized datasets, which are difficult to obtain. This paper aims to bridge this gap by directly using facial expressions within an image to inform the captioning process, without relying on external sentiment resources.
Methods
The core of the Face-Cap system involves two main components: a facial expression recognition (FER) model and an image captioning model.
- Facial Expression Recognition (FER): The authors train a VGGnet-based model on the FER-2013 dataset, which contains "in-the-wild" images with labeled emotions. This model achieves a respectable 72.7% accuracy, outperforming human performance on the same dataset. This trained FER model is then used as a feature extractor to obtain probabilities for seven universal emotions (happiness, sadness, fear, surprise, anger, disgust, neutral) from faces detected in the images. These probabilities are then converted into a one-hot encoded vector representing the aggregate facial emotion of the image.
- Image Captioning Model (Face-Cap): The captioning model is based on a Long Short-Term Memory (LSTM) network, similar to state-of-the-art attention-based models. The key innovation is how facial expression features are integrated:
- Face-CapF: This variant feeds the facial expression features into the LSTM at every time step, alongside image features and previous word embeddings. It also incorporates a "face loss function" during training to encourage the model to learn from these facial cues.
- Face-CapL: This variant is similar but removes the facial features from being fed at every time step. Instead, it uses them to initialize the LSTM's memory and hidden states, allowing the model to "forget" or ignore irrelevant facial information during generation.
The image features themselves are extracted using an Oxford VGGnet pre-trained on ImageNet and are weighted using an attention mechanism, allowing the model to focus on relevant image regions for each word it generates.
Datasets
- FER-2013: Used for training the facial expression recognition model. It contains 35,887 images with labels for six basic emotions plus neutral.
- FlickrFace11K: A newly created dataset for training and evaluating the image captioning models. It's a subset of the Flickr 30K dataset, specifically filtered to include images containing human faces. It comprises approximately 11,000 image-caption pairs. This dataset is made publicly available.
Key Insights and Ideas
- Emotional Context Enhances Action Descriptions: Surprisingly, the improved caption quality from Face-Cap models didn't stem from adding emotional adjectives (e.g., "happy," "sad"). Instead, the models generated captions with a greater variety of described actions. This suggests that understanding the emotional state of individuals in an image can lead to a richer understanding of what they are doing.
- Facial Features as a Rich Source of Information: Directly using facial expressions, rather than relying on external sentiment lexicons, proves effective. This approach captures a more nuanced and direct emotional signal from the image content.
- Selective Use of Emotional Cues is Beneficial: The Face-CapL model, which can selectively ignore facial features when irrelevant, performs slightly better than Face-CapF, which forces the model to consider facial features at every step. This highlights the importance of allowing the model flexibility in how it utilizes emotional information.
- Novel Dataset for Emotion-Aware Captioning: The creation and release of FlickrFace11K provide a valuable resource for future research in this specific area.
Key Results
- Outperformance of Baseline: Both Face-CapF and Face-CapL models consistently outperformed the state-of-the-art baseline (Xu's model) across standard evaluation metrics like BLEU, ROUGE-L, METEOR, CIDEr, and SPICE.
- Face-CapL Leads: Face-CapL generally achieved the best results, particularly in metrics like CIDEr, which penalizes generic captions.
- Action Diversity: Linguistic analysis revealed that Face-Cap models generated captions with a higher entropy in verb distributions, indicating more varied action descriptions. Verbs like "smiling," "singing," and "looking" were more frequently and appropriately generated by the Face-Cap models.
Limitations and Future Work
- Limited Emotion Set: The current models are trained on a fixed set of seven basic emotions. Future work could explore a richer set of emotions, including confusion and curiosity.
- Potential for Misinterpretation: The paper acknowledges instances where facial features might be misinterpreted due to visual likeness (e.g., mistaking a visual cue for "holding a microphone" when it's not).
- Soft Injection of Features: The authors suggest exploring alternative architectures for injecting facial emotions, such as a "soft injection" approach, which might offer more nuanced integration.
- Improving Emotion Recognition: Developing more robust facial expression recognition models for diverse "in-the-wild" scenarios could further enhance captioning performance.
Summary updated