TL;DR
Dixit is an interactive visual storytelling system that breaks the "black box" of end-to-end deep learning by introducing a human-editable intermediate layer of text terms. By allowing users to manipulate the specific objects (nouns) and actions (verb frames) extracted from a photo sequence, the system enables controllable, interpretable, and collaborative narrative generation.
What Problem Does This Paper Address?
Visual storytelling—generating a cohesive narrative from a sequence of images—is significantly more complex than simple image captioning. While a caption describes what is in a photo, a story requires imagination, context, and subjective flair.
The authors identify four major hurdles in current state-of-the-art models:
- Lack of Interpretability: Most models are end-to-end deep neural networks (DNNs). It is nearly impossible to tell why a model chose a specific plot point based on a specific image feature.
- Inability to Modify: In a standard DNN, the same input sequence always yields the same story. Users have no "knobs" to turn if they want to change the narrative direction.
- Evaluation Difficulty: Automatic metrics like BLEU or METEOR are notoriously poor at capturing the "soul" or quality of a creative story.
- Data Scarcity: High-quality datasets pairing image sequences with rich, human-written stories are expensive and difficult to produce at scale.
Methods
Dixit tackles these problems using a two-stage framework that separates visual recognition from linguistic composition, linked by a "term manipulation" interface.
- Stage 1: Image-to-Term Prediction: The system uses a pre-trained ResNet-152 to extract visual features from each image. These features are fed into an LSTM that predicts a set of "terms." These aren't just random words; they are categorized into nouns (objects) and verb frames (actions) derived from FrameNet, a lexical database of semantic relationships.
- The Interaction Layer: This is the core innovation. Before a story is generated, the predicted terms are shown to the user. The user can delete inaccurate terms, add new ones via a search engine, or swap them to change the story's focus (e.g., changing "man" to "boy" to shift the protagonist).
- Stage 2: Term-to-Story Generation: The final terms are fed into a Sentence-aware Transformer. Unlike a standard Transformer, this model uses two types of positional embeddings:
- Inter-sentence (InterPE): Indicates the order of images/sentences in the sequence (e.g., "this is the 3rd image").
- Intra-sentence (IntraPE): Indicates the position of a word within a specific sentence. This allows the model to maintain global coherence across the entire five-photo arc.
Datasets
The modularity of Dixit allows the authors to train the two stages on different, more abundant datasets rather than relying solely on rare image-story pairs:
- MS-COCO: Used to train the Stage 1 term predictor, leveraging its vast library of images and descriptive captions.
- ROCStories: A corpus of 5-sentence commonsense stories used to train the Stage 2 story generator. To bridge the gap, the authors used a "back-translation" style technique to extract terms from these stories to create term-story training pairs.
- FrameNet: Used to provide a structured semantic framework for actions and events.
Key Insights and Ideas
- The "Term" as a Bridge: By using text terms as an intermediate representation, the authors create a "white box" moment. The user sees exactly what the AI sees. If the AI sees a "motorcycle" but the user wants a story about a "bicycle," the user can intervene before the error propagates into the final text.
- Decoupling Vision and Language: Because the system is two-stage, the story generator can be trained on massive text-only corpora (like ROCStories). This bypasses the "data scarcity" bottleneck of traditional visual storytelling.
- Verb Frames vs. Simple Verbs: By using FrameNet, the system captures the context of an action (e.g., the frame "Seeking" includes lexical units like "watch" or "look for"), leading to more nuanced storytelling than simple label-matching.
Key Results
The paper demonstrates the system's utility through qualitative examples of "Term Manipulation." In one instance, the initial model predicted "man" and "sitting." By manually changing the terms to "boy" and "sleep," the user successfully steered the AI to generate a completely different, more desirable narrative ("the boy went back to his seat to sleep").
Quantitatively, the authors emphasize that the system acts as an auto-labeling tool. As users interact with Dixit—correcting terms and rating stories—they generate a high-quality, human-in-the-loop dataset that can be used to further refine future storytelling models.
Limitations and Future Work
- Subjectivity of Terms: While nouns and verb frames cover a lot of ground, they might miss abstract concepts or emotions that are central to storytelling.
- User Effort: The system relies on the user being willing to iterate. If the initial term predictions are too far off, the manual correction process might become tedious.
- Evaluation: The paper focuses on the system architecture and interaction; a large-scale human study comparing Dixit's output to end-to-end models would be a logical next step to prove superior story quality.
Summary updated