Multimodal Storytelling via Generative Adversarial Imitation Learning

Zhiqian Chen, Xuchao Zhang, Arnold P. Boedihardjo, Jing Dai, Chang-Tien Lu

Abstract

This paper introduces MIL-GAN, a framework that utilizes Generative Adversarial Imitation Learning (GAIL) to automatically generate event storylines—chains of connected entities—from large-scale multimodal data. By "imitating" how humans connect dots across text and images, the model moves beyond rigid, manually-defined metrics to capture the subtle, multimodal patterns that make a story meaningful. Experimental results demonstrate that MIL-GAN outperforms traditional methods in both specificity and user preference.

Introduction

In an era of information overload, search engines excel at finding isolated facts but struggle to weave them into a coherent "story." A story, in this context, is a sequence of events or entities that explain a complex situation, such as the aftermath of a terrorist attack or a political election. Existing storytelling algorithms often rely on hand-crafted coherence metrics that fail to capture human intent and typically focus on a single data modality, missing the cross-modal links essential for resolving ambiguity.

Conceptual overview of shared patterns in multimodal storytelling across different events.

The MIL-GAN Framework

The authors propose Multimodal Imitation Learning via Generative Adversarial Networks (MIL-GAN). Rather than pre-defining a "good" story, the model learns a reward function directly from expert human examples through an adversarial process.

Architecture

  • The Generator (Policy): An LSTM-based model that selects the next entity in a storyline. Its objective is to generate sequences that are indistinguishable from human-made stories to the discriminator.
  • The Discriminator (Reward Function): A TextCNN that evaluates storylines against human-provided ground truth. It provides the reward signal used to optimize the generator via Reinforcement Learning (specifically the REINFORCE algorithm).
  • Multimodal Embedding: To bridge text and images, the framework employs an SVD-based semantic embedding. This maps images into a "sentential space," allowing the model to understand, for instance, that a photo of a building and the word "victim" are contextually linked.
The MIL-GAN architecture showing the interaction between multimodal generators and discriminators.

Experimental Setup and Datasets

To address the lack of standard benchmarks, the authors curated a new Multimodal Storytelling Dataset consisting of:

  • Training Set: 1,310 storylines related to "Homicide" events and 934 related to "Protest" events (e.g., 9/11, Charlie Hebdo).
  • Test Set: Unseen events such as the disappearance of Malaysia Airlines Flight 370 and the 2014 Iguala mass kidnapping.
  • Source Data: Documents and images scraped from Google News and Wikipedia.

Key Results and Insights

  • Imitation over Definition: By using Inverse Reinforcement Learning (IRL), the model discovers latent connectivity features that humans find interesting, avoiding the pitfalls of subjective math formulas.
  • Multimodality as a De-noiser: Anchoring images to textual space allows the model to distinguish relevant visual information (e.g., identifying a building as a "target") from background noise.
  • User Preference: In head-to-head comparisons on Amazon Mechanical Turk, users preferred MIL-GAN's stories 31.4% of the time, significantly outperforming text-only models (28.1%) and standard Policy Gradient methods (17.3%).
  • Specificity and Transferability: MIL-GAN generated more specific entity names (e.g., "Ayotzinapa") compared to generic baselines. Furthermore, it successfully transferred storytelling patterns from one domain (e.g., 9/11) to entirely different events (e.g., MH370).

Conclusion

MIL-GAN represents a significant step toward automated, human-like storytelling. By combining imitation learning with multimodal embeddings, the framework captures narrative nuances that traditional metrics miss. Future work will explore more complex structures, such as story graphs, to capture branching narratives beyond linear chains.

Summary updated

BibTeX
@inproceedings{chen2017multimodal,
  author = {Zhiqian Chen and Xuchao Zhang and Arnold P. Boedihardjo and Jing Dai and Chang-Tien Lu},
  title = {{Multimodal Storytelling via Generative Adversarial Imitation Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/554},
}
Figure