Overview of the Automated Story Illustration Task at FIRE 2015

Debasis Ganguly, Iacer Calixto, Gareth J. F. JonesFIRE Workshops 2015

Introduction

The FIRE 2015 workshop introduced a benchmark and shared task for Automated Story Illustration. This research provides a dataset based on Aesop’s Fables and evaluates systems designed to retrieve relevant images from a static collection to visually "expand" text, making it more engaging for young readers.

The Challenge of Multimodal Expansion

Unlike standard search where users provide explicit queries, story illustration requires the system to treat the text as an implicit query. The system must identify "agents, events, and actions" within the narrative and decide which concepts are worth illustrating.

Figure 4: A composite of images illustrating various concepts from a story, demonstrating the goal of the automated illustration task.

Methodology

The authors framed the task as an Information Retrieval (IR) problem with three main steps:

  1. Concept Extraction: Identifying key phrases (e.g., "Ant" or "summer day") that require illustration.
  2. Query Formulation: Turning those concepts into search terms.
  3. Image Retrieval & Merging: Retrieving ranked lists of images for each concept and merging them into a single coherent list.

The study compared two baseline approaches—unweighted "bag of words" and tf-idf weighted keywords—against external submissions using word embeddings and the Divergence from Randomness (DFR) model.

Dataset and Evaluation

The study used 22 stories from Aesop’s Fables and the ImageCLEF 2010 Wikipedia image collection. Relevance judgments were made on a 5-point scale by human annotators to ensure the "intent" of the illustration was preserved.

Figure 5: An example of a retrieved image for the concrete concept 'Grasshopper'.

Key Results: The "Simplicity Paradox"

The evaluation revealed a surprising "simplicity paradox": the authors' basic tf-idf weighted baseline significantly outperformed more complex models.

  • Top Performer: The tf-idf baseline achieved a MAP of 0.0529 and a P@5 of 0.1714.
  • Participant Performance: Models using word embeddings (MAP 0.0107) and DFR (MAP 0.0053) struggled to match the baseline.

This suggests that in niche domains like fables, general-purpose word embeddings may introduce "noise" that leads the search away from specific, concrete imagery.

Conclusion

The low overall MAP scores indicate that automated story illustration remains a difficult task. Future work should focus on better query representation and more sophisticated NLP to identify holistic "meaningful units" rather than isolated keywords.

Summary updated

BibTeX
@inproceedings{ganguly2015overview,
  author = {Debasis Ganguly and Iacer Calixto and Gareth J. F. Jones},
  title = {{Overview of the Automated Story Illustration Task at FIRE 2015}},
  booktitle = {FIRE Workshops},
  year = {2015},
}
Figure