Abstract
This paper introduces MIL-GAN, a framework that utilizes Generative Adversarial Imitation Learning (GAIL) to automatically generate event storylines—chains of connected entities—from large-scale multimodal data. By "imitating" how humans connect dots across text and images, the model moves beyond rigid, manually-defined metrics to capture the subtle, multimodal patterns that make a story meaningful. Experimental results demonstrate that MIL-GAN outperforms traditional methods in both specificity and user preference.
Introduction
In an era of information overload, search engines excel at finding isolated facts but struggle to weave them into a coherent "story." A story, in this context, is a sequence of events or entities that explain a complex situation, such as the aftermath of a terrorist attack or a political election. Existing storytelling algorithms often rely on hand-crafted coherence metrics that fail to capture human intent and typically focus on a single data modality, missing the cross-modal links essential for resolving ambiguity.
The MIL-GAN Framework
The authors propose Multimodal Imitation Learning via Generative Adversarial Networks (MIL-GAN). Rather than pre-defining a "good" story, the model learns a reward function directly from expert human examples through an adversarial process.
Architecture
- The Generator (Policy): An LSTM-based model that selects the next entity in a storyline. Its objective is to generate sequences that are indistinguishable from human-made stories to the discriminator.
- The Discriminator (Reward Function): A TextCNN that evaluates storylines against human-provided ground truth. It provides the reward signal used to optimize the generator via Reinforcement Learning (specifically the REINFORCE algorithm).
- Multimodal Embedding: To bridge text and images, the framework employs an SVD-based semantic embedding. This maps images into a "sentential space," allowing the model to understand, for instance, that a photo of a building and the word "victim" are contextually linked.
Experimental Setup and Datasets
To address the lack of standard benchmarks, the authors curated a new Multimodal Storytelling Dataset consisting of:
- Training Set: 1,310 storylines related to "Homicide" events and 934 related to "Protest" events (e.g., 9/11, Charlie Hebdo).
- Test Set: Unseen events such as the disappearance of Malaysia Airlines Flight 370 and the 2014 Iguala mass kidnapping.
- Source Data: Documents and images scraped from Google News and Wikipedia.
Key Results and Insights
- Imitation over Definition: By using Inverse Reinforcement Learning (IRL), the model discovers latent connectivity features that humans find interesting, avoiding the pitfalls of subjective math formulas.
- Multimodality as a De-noiser: Anchoring images to textual space allows the model to distinguish relevant visual information (e.g., identifying a building as a "target") from background noise.
- User Preference: In head-to-head comparisons on Amazon Mechanical Turk, users preferred MIL-GAN's stories 31.4% of the time, significantly outperforming text-only models (28.1%) and standard Policy Gradient methods (17.3%).
- Specificity and Transferability: MIL-GAN generated more specific entity names (e.g., "Ayotzinapa") compared to generic baselines. Furthermore, it successfully transferred storytelling patterns from one domain (e.g., 9/11) to entirely different events (e.g., MH370).
Conclusion
MIL-GAN represents a significant step toward automated, human-like storytelling. By combining imitation learning with multimodal embeddings, the framework captures narrative nuances that traditional metrics miss. Future work will explore more complex structures, such as story graphs, to capture branching narratives beyond linear chains.
Summary updated