Causal-Story: Local Causal Attention Utilizing Parameter-Efficient Tuning For Visual Story Synthesis

Tianyi Song, Jiuxin Cao, Kun Wang, Bo Liu, Xiaofeng Zhang

TL;DR

This paper introduces Causal-Story, a novel approach to visual story synthesis that improves global consistency by incorporating a local causal attention mechanism. This allows the model to better understand and leverage the causal relationships between previous story elements and the current frame, leading to more coherent and visually appealing generated stories.

What Problem Does This Paper Address?

Generating coherent visual narratives, like comic strips or storyboards, from text descriptions is a challenging task. Existing methods often treat all previous frames and captions equally when generating a new frame, failing to recognize that not all historical information is equally relevant to the current step. This can lead to inconsistencies and a loss of narrative flow. For instance, a detail from an early frame might be crucial for a later frame, while another intermediate detail might be entirely irrelevant. The paper aims to address this by enabling the model to selectively focus on the most causally relevant past information.

Methods

Causal-Story builds upon the latent diffusion model (LDM) architecture, specifically inspired by AR-LDM. The core innovation is the Local Causal Attention Mask (LCAM). Instead of a uniform attention mechanism that flattens all previous context, LCAM introduces a mask that explicitly models the causal relationships between previous captions, frames, and the current caption. This allows the model to assign different weights to historical information based on its perceived relevance to the current frame generation.

To make the model more efficient, Causal-Story also incorporates a parameter-efficient tuning adapter. This lightweight module fine-tunes a pre-trained model without requiring extensive retraining, significantly reducing training time and computational resources while maintaining effectiveness.

The underlying diffusion process involves a forward process that adds noise to data and a reverse process where a UNet learns to denoise the data, guided by text and image conditions. Causal-Story enhances this by integrating the LCAM within the UNet's attention layers.

Datasets

The paper evaluates Causal-Story on two existing datasets:

  • PororoSV: A dataset for story visualization.
  • FlintstonesSV: Another dataset for story visualization.

No new datasets were introduced in this work.

Key Insights and Ideas

  • Causality Matters in Storytelling: The most significant insight is that not all historical context is equally important for generating the next frame in a story. Explicitly modeling causal relationships leads to better narrative coherence.
  • Local Attention for Global Consistency: By focusing attention locally and causally, the model can avoid being misled by irrelevant past information, thereby improving the global consistency of the generated story.
  • Efficiency Through Adapters: Parameter-efficient tuning with adapters is a viable strategy for adapting large diffusion models to new tasks like story synthesis without prohibitive computational costs.
  • Mitigating Temporal Inconsistency: The causal attention mask helps address the problem of quality degradation and temporal inconsistency that can arise in long or complex narratives.

Key Results

Causal-Story achieved state-of-the-art FID scores on both the PororoSV and FlintstonesSV datasets for story visualization and continuation tasks. For example, on PororoSV for story visualization, Causal-Story achieved an FID score of 16.28, outperforming previous methods like AR-LDM (16.89) and StoryDALL-E (65.61).

Qualitatively, the generated frames demonstrate improved storytelling and visual coherence. The paper highlights instances where previous models failed to correctly interpret context (e.g., misinterpreting "Pororo's house" or ignoring characters like "Poby"), which Causal-Story successfully addresses. Furthermore, the adapter mechanism led to improved training and sampling speeds compared to AR-LDM, with Causal-Story achieving faster training (65h 31m 38s vs. 71h 43m 54s) and sampling (58h 27m 21s vs. 59h 04m 32s) for the same number of epochs.

Limitations and Future Work

The paper acknowledges that while the causal attention mask helps mitigate issues with irrelevant captions, longer captions can still pose challenges. The authors suggest that future work could explore further improvements in handling complex and lengthy textual descriptions. Additionally, while the adapter provides efficiency, further research into optimizing parameter-efficient tuning for diffusion models in story generation could be beneficial. The paper also implicitly suggests that exploring more sophisticated ways to quantify and incorporate causal relationships could lead to even more robust story synthesis.

Summary updated

Abstract

The excellent text-to-image synthesis capability of diffusion models has driven progress in synthesizing coherent visual stories. The current state-of-the-art method combines the features of historical captions, historical frames, and the current captions as conditions for generating the current frame. However, this method treats each historical frame and caption as the same contribution. It connects them in order with equal weights, ignoring that not all historical conditions are associated with the generation of the current frame. To address this issue, we propose Causal-Story. This model incorporates a local causal attention mechanism that considers the causal relationship between previous captions, frames, and current captions. By assigning weights based on this relationship, Causal-Story generates the current frame, thereby improving the global consistency of story generation. We evaluated our model on the PororoSV and FlintstonesSV datasets and obtained state-of-the-art FID scores, and the generated frames also demonstrate better storytelling in visuals.
BibTeX
@inproceedings{song2024causal,
  author = {Tianyi Song and Jiuxin Cao and Kun Wang and Bo Liu and Xiaofeng Zhang},
  title = {{Causal-Story: Local Causal Attention Utilizing Parameter-Efficient Tuning For Visual Story Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=NdTUu0CazZ},
}
Figure