TL;DR
This paper introduces a novel "Multimodal Event Transformer" (MET) for generating story endings that are both coherent with a given plot and relevant to an accompanying image. By leveraging event-centric reasoning on visual and semantic graphs, MET effectively mines implicit information, leading to state-of-the-art performance in image-guided story ending generation.
What Problem Does This Paper Address?
Generating a compelling story ending is a challenging task in natural language generation. Simply relying on the story plot often leads to generic or inane endings because the model struggles to infer implicit information. The "image-guided story ending generation" (IgSEG) task aims to improve this by incorporating an ending image. However, existing methods primarily focus on fusing features from the plot and image, often overlooking the crucial step of reasoning and mining deeper, implicit information within each modality. This can result in endings that are not specific or informative enough, failing to fully leverage the multimodal context.
Methods
The core of this paper is the Multimodal Event Transformer (MET). Its key components are:
- Event Graph Construction:
- Semantic Event Graphs: Story plots are parsed using Semantic Role Labeling (SRL) to extract events (predicate-argument structures). These events are then organized into a graph where nodes represent events and sentences, and edges capture relationships.
- Visual Event Graphs: Ending images are processed by a scene graph parser to identify objects, their attributes, and relationships. This forms a visual event graph.
- Event-based Reasoning: Relational Graph Convolutional Networks (RGCN) are employed to reason over both semantic and visual event graphs. This allows the model to infer implicit information within each modality by propagating messages along different relation types.
- Cross-modal Fusion: The semantic and visual event graphs are merged. RGCN is used again to fuse these cross-modal features, aiming to complement information between modalities while preserving graph structures.
- Multimodal Injector: A novel component that adaptively extracts key information from both visual and semantic features based on the decoder's current hidden state. This injector then fuses this information and passes it to the decoder, ensuring that relevant multimodal context is incorporated at each generation step.
- Incoherence Detection: An auxiliary training objective designed to improve the model's understanding of story plot context and the robustness of its graph modeling. It works by randomly perturbing sentence nodes in the semantic event graph and training the model to detect these changes.
- Story Ending Generation: A standard Transformer decoder is used to generate the story ending, conditioned on the fused multimodal information from the injector.
Datasets
The paper evaluates its method on two standard datasets for image-guided story ending generation:
- VIST-Ending (VIST-E): A dataset derived from the VIST dataset, comprising 39,920 training, 4,963 validation, and 5,030 testing samples.
- LSMDC-Ending (LSMDC-E): A dataset with 20,151 training, 1,477 validation, and 2,005 testing samples, collected from LSMDC 2021.
Additionally, the Visual Genome dataset was used to train the visual relationship detector for constructing visual event graphs.
Key Insights and Ideas
- Event-centric reasoning is crucial for implicit information: The paper strongly advocates that moving beyond simple feature fusion to explicitly reasoning about events and their relationships within each modality (text and image) is key to generating more informative and specific story endings.
- Graph structures capture richer relationships: Representing events and visual concepts as graphs allows the model to capture complex interdependencies that might be lost in flattened feature vectors.
- Adaptive multimodal injection matters: The multimodal injector's ability to selectively attend to and fuse relevant information from both modalities based on the decoder's state is vital for effective cross-modal integration.
- Incoherence detection improves robustness: The auxiliary task of detecting sentence perturbations in the semantic event graph helps the model better understand plot context and makes its graph reasoning more robust.
Key Results
The proposed Multimodal Event Transformer (MET) achieves state-of-the-art performance on both the VIST-E and LSMDC-E datasets across various metrics, including BLEU@n, METEOR, ROUGE-L, and CIDEr. For instance, on the VIST-E test set, MET achieved the highest rSUM score of 107.82, outperforming strong baselines like MMT (88.41) and MGCL (82.89). Ablation studies confirm the effectiveness of each proposed component, with significant performance drops observed when event-based reasoning, cross-modal fusion, or the multimodal injector were removed.
Limitations and Future Work
The authors acknowledge a limitation in their current cross-modal fusion approach: it connects whole image nodes to whole sentence nodes. They suggest that a more fine-grained connection between semantic events and visual objects is needed. Future work could explore how to pass this finer-grained information between visual and semantic event graphs to further enhance multimodal integration.
Summary updated