Detecting and Grounding Important Characters in Visual Stories

Danyang Liu, Frank KellerAAAI 2023 · DOI

TL;DR

This paper introduces VIST-Character, a new dataset for visual storytelling that annotates characters, their co-reference chains, and importance ratings. They also propose two new tasks: important character detection and character grounding in visual stories, and present baseline models for these tasks. This work aims to shift visual storytelling from an object-centric to a character-centric perspective, enabling more coherent and engaging narratives.

What Problem Does This Paper Address?

Current visual storytelling models excel at generating descriptive text for image sequences but often fail to capture the narrative's core – its characters. These models treat all detected objects equally, leading to stories that describe events but lack consistent characters, proper co-reference resolution, and a sense of character importance (e.g., distinguishing protagonists from minor figures). This limitation stems from a lack of datasets with rich character-centric annotations, making it difficult to develop models that can truly understand and generate character-driven narratives.

Methods

The authors propose a methodology that breaks down the problem into four sub-tasks:

  1. Textual Character Detection and Co-reference: They use part-of-speech tagging and WordNet to identify potential character mentions in text, followed by pre-trained co-reference resolution tools (NeuralCoref and SpanBERT) to group mentions into chains. They also specifically handle plural and group characters.
  2. Visual Character Detection and Co-reference: For images, they use MTCNN for face detection and then employ either Inception ResNet or CLIP for feature extraction. K-means clustering is used to group visual appearances into co-reference chains.
  3. Alignment of Textual and Visual Co-reference Chains (Character Grounding): This is framed as a bipartite graph matching problem. They propose two similarity measures:
  4. Distributional Similarity: This method represents character distributions across sentences/images as binary vectors and matches chains based on their overlap.
  5. CLIP-based Similarity: This approach leverages the pre-trained CLIP model to measure the similarity between textual mentions and visual appearances within chains. The Kuhn-Munkres algorithm is then used to find the optimal alignment.

  6. Importance Ranking: They hypothesize that character importance correlates with frequency. They propose a count-based importance ranking, where the length of a character's multi-modal co-reference chain serves as its importance score.

Datasets

  • VIST-Character: This is a newly introduced dataset, augmenting the test set of the existing VIST (Visual Storytelling) dataset. It includes character co-reference chains (textual and visual) and importance ratings for characters across 770 visual stories.

Key Insights and Ideas

  • Character-centricity is Crucial for Storytelling: The paper strongly argues that moving beyond object detection to focus on characters is essential for generating truly compelling and coherent visual stories.
  • Visual Grounding is Harder than Traditional Language Grounding: Aligning textual character mentions with visual bounding boxes in image sequences is challenging due to the discrete nature of images and the many-to-many mapping required for co-reference chains.
  • Frequency as a Proxy for Importance: The assumption that more frequent characters are more important is validated by a positive correlation between occurrence counts and human-assigned importance ratings.
  • Textual Information Dominates Importance Ranking: The text-based model for identifying the most important character (protagonist) significantly outperforms image-only or multi-modal approaches, likely due to the inherent noise in image detection and the direct link between text frequency and human ratings.
  • Plural/Group Characters Add Complexity: While important for a complete understanding, handling plural and group characters poses challenges for grounding and can sometimes negatively impact the precision of identifying the single most important character.

Key Results

  • Character Detection: Textual character detection achieved higher precision and recall (74.4% P, 90.3% R) compared to image sequence detection (40.5% P, 69.1% R), with the latter being hampered by background characters.
  • Character Co-reference: Textual co-reference resolution generally outperformed visual co-reference resolution. SpanBERT was found to be superior for text, and CLIP for visual co-reference.
  • Character Grounding: The alignment algorithm performed significantly better when using gold-standard annotations for character detection and co-reference, highlighting its sensitivity to input errors. CLIP-based similarity did not outperform distributional similarity, possibly due to the generic nature of some textual mentions.
  • Importance Ranking: Text-only models achieved the highest precision (P@5 = 87.4%) for identifying important characters, reinforcing the idea that textual frequency is a strong indicator. Multi-modal approaches showed potential but were hindered by noise from visual detection.

Limitations and Future Work

  • Sensitivity to Input Errors: The character grounding performance is heavily reliant on the accuracy of preceding character detection and co-reference steps. Errors in these stages significantly degrade grounding results.
  • CLIP's Performance: CLIP-based similarity did not outperform simpler distributional methods for grounding, suggesting that its powerful image-text understanding might be less effective with the specific type of generic language found in VIST stories.
  • Multi-modal Importance Ranking: The current multi-modal approach for importance ranking was found to be negatively impacted by visual noise, suggesting a need for better integration of textual and visual cues.
  • Future Work: The authors plan to use their dataset and models to build character-centric visual storytelling models, extend them to more complex narratives like comic books, and release the dataset and codebase. They also aim to explore more sophisticated multi-modal integration for importance ranking.

Summary updated

BibTeX
@inproceedings{liu2023detecting,
  author = {Danyang Liu and Frank Keller},
  title = {{Detecting and Grounding Important Characters in Visual Stories}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26551},
}
Figure