Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools

Fariba Lotfi, Amin Beheshti, Helia Farhood, Matineh Pooshideh, Mansour Jamzad, Hamid BeigyAlgorithms 2023 · DOI · OpenReview

TL;DR

This paper presents a systematic review of methods and tools for "storytelling with image data," aiming to transform raw images into coherent narratives. It proposes a framework for this process, highlighting the challenges and opportunities in bridging the gap between visual information and human understanding.

What Problem Does This Paper Address?

In our data-rich world, vast amounts of information are locked within images and videos. While image processing and analytics have advanced, effectively extracting, linking, and comprehending this visual data to create a cohesive narrative remains a significant challenge. Current methods often struggle to go beyond simple labeling or captioning, failing to construct a story that captures the temporal, contextual, and subjective aspects of events depicted in images. This gap hinders applications ranging from investigative analysis to richer user experiences.

Methods

The authors conducted a systematic literature review to identify and analyze existing methods and tools for storytelling with image data. They propose a framework encompassing several phases:

  • Data Curation: Cleaning, preprocessing, extraction, enrichment, and linking of raw image data.
  • Story Modeling: Transforming curated data into contextualized knowledge, often using Knowledge Graphs, to discover events and entities.
  • Intelligent Narrative Discovery: Identifying user intent and summarizing image collections to inform story generation.
  • Story Creation: Generating narratives using various techniques, including Reinforcement Learning (RL) and Generative Adversarial Networks (GANs).
  • Story Presentation and Querying: Visualizing the generated stories and enabling user interaction.
  • Quality Assessment: Evaluating the generated stories using both automatic metrics and human judgment.

The paper categorizes techniques within each phase, discussing traditional computer vision approaches alongside modern machine learning and deep learning methods.

Datasets

The paper discusses several datasets used in the field of visual storytelling:

  • VIST (Visual Storytelling Dataset): A sequential vision-to-language dataset comprising Flickr albums with images and descriptions, used for training and evaluating storytelling models.
  • VIST-Edit1: A subset of VIST with human-edited versions of machine-generated stories.
  • AESOP: A dataset built on principles of creativity, causal narratives, and world knowledge, featuring image-text panels.
  • VHED (VIST Human Evaluation Data): A dataset derived from VIST, focusing on human evaluation of stories.
  • Cooking Dataset: An image-based dataset for sequential procedural text generation, containing recipes and associated photos.
  • Recipe1M+: A large-scale corpus of cooking recipes and food images.

Key Insights and Ideas

  • Storytelling as a Metaphor: The paper argues that "storytelling" is a powerful metaphor for image data analysis, enabling the capture and representation of narratives and insights hidden within interconnected data.
  • Beyond Captioning: Storytelling with image data is distinct from simple image/video captioning. It involves subjective interpretation, narrative structure, and creating a coherent sequence of events, often from multiple images.
  • The Importance of Context: Enriching extracted image features with external knowledge bases (like Knowledge Graphs) is crucial for generating meaningful and context-aware narratives.
  • Subjectivity and Multiple Perspectives: Stories can be told from various perspectives, ranging from abstract to detailed, reflecting the subjective nature of human interpretation.
  • Bridging the Visual Gap: A key challenge is filling the visual gaps between images with a subjective story, creating a seamless and engaging narrative.

Key Results

The paper is a systematic review and does not present novel experimental results. Instead, it synthesizes findings from existing research, categorizing methods and tools across different tasks of the storytelling pipeline. It highlights the progress made in areas like object detection, image captioning, and narrative generation, while also identifying significant challenges.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Lack of Formal Framework: The field is still in its early stages, lacking a formal framework for story modeling and creation from data.
  • Technical Challenges: Difficulties persist in labeling, captioning, and extracting information from images accurately and efficiently.
  • Bias and Subjectivity: The subjective nature of storytelling can introduce bias into narratives, requiring careful consideration.
  • Scalability: Handling the sheer volume of image data poses significant challenges for processing, storage, and presentation.
  • Quality Assessment: Developing robust and automated methods for assessing the quality of generated stories remains a complex challenge, often relying on human judgment.

Future work suggested includes developing more advanced metrics for story evaluation, exploring creative generation approaches (e.g., GANs), integrating multimodal data sources, creating new interactive tools, and applying storytelling techniques to new domains like education and healthcare.

Summary updated

Abstract

In our digital age, data are generated constantly from public and private sources, social media platforms, and the Internet of Things. A significant portion of this information comes in the form of unstructured images and videos, such as the 95 million daily photos and videos shared on Instagram and the 136 billion images available on Google Images. Despite advances in image processing and analytics, the current state of the art lacks effective methods for discovering, linking, and comprehending image data. Consider, for instance, the images from a crime scene that hold critical information for a police investigation. Currently, no system can interactively generate a comprehensive narrative of events from the incident to the conclusion of the investigation. To address this gap in research, we have conducted a thorough systematic literature review of existing methods, from labeling and captioning to extraction, enrichment, and transforming image data into contextualized information and knowledge. Our review has led us to propose the vision of storytelling with image data, an innovative framework designed to address fundamental challenges in image data comprehension. In particular, we focus on the research problem of understanding image data in general and, specifically, curating, summarizing, linking, and presenting large amounts of image data in a digestible manner to users. In this context, storytelling serves as an appropriate metaphor, as it can capture and depict the narratives and insights locked within the relationships among data stored across different islands. Additionally, a story can be subjective and told from various perspectives, ranging from a highly abstract narrative to a highly detailed one.
BibTeX
@article{lotfi2023storytelling,
  author = {Fariba Lotfi and Amin Beheshti and Helia Farhood and Matineh Pooshideh and Mansour Jamzad and Hamid Beigy},
  title = {{Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools}},
  journal = {Algorithms},
  year = {2023},
  doi = {10.3390/a16030135},
}
Figure