Abstract
Generating long-form videos with high narrative consistency remains a significant challenge in AI research. This paper presents MemFlow, a novel framework that utilizes a dynamic, "flowing" memory mechanism for streaming text-to-video generation. By leveraging semantic retrieval guided by text prompts and employing sparse activation, MemFlow maintains visual coherence over extended durations (60+ seconds) while achieving a near-real-time generation speed of 18.7 FPS.
Introduction
Current text-to-video (T2V) systems often struggle with the "memory-efficiency trade-off." When generating long videos in chunks, models must remember previous context to ensure characters and settings do not "drift." Traditional approaches either use a fixed window of recent frames—which risks forgetting early narrative elements—or compress past frames into a small set of tokens, leading to a loss of fine-grained detail. MemFlow addresses these limitations by treating memory as an adaptive bank that retrieves relevant historical context based on the evolving story.
Methodology
The MemFlow architecture is built upon two core innovations designed to balance narrative depth with computational efficiency.
Narrative Adaptive Memory (NAM)
NAM replaces static memory windows with a Semantic Retrieval mechanism. Instead of simply looking at the most recent frames, the model uses the text prompt for the current video chunk to query a memory bank containing visual Key-Value (KV) caches of all previously generated frames.
By calculating attention scores between text tokens and historical visual tokens, NAM pulls the most semantically relevant context. To maintain efficiency, MemFlow employs a "Redundant Removal" heuristic, condensing 5-second chunks into single "prototype" frames, which captures the essential visual information without overwhelming the system.
Sparse Memory Activation (SMA)
To prevent the large memory bank from slowing down the attention mechanism, MemFlow utilizes Sparse Memory Activation. SMA strategically activates only the top-k most relevant tokens in the memory bank for each query. This approach significantly reduces computational overhead, allowing the model to maintain high performance even as the video length increases.
Experimental Results
MemFlow was evaluated using the Wan2.1-T2V-1.3B architecture on the LongLive Switch Prompt Dataset and the VBench-Long benchmark.
Performance and Efficiency
- Speed: MemFlow operates at 18.7 FPS on a single NVIDIA H100, representing a 38x speedup over competitors like SkyReels-V2.
- Visual Quality: On VBench-Long, MemFlow achieved a Quality Score of 85.02 and an Aesthetic Score of 61.07, outperforming other long-form models such as LongLive and FramePack.
- Consistency: Qualitative analysis shows that MemFlow effectively prevents "character drifting" and the generation of redundant subjects, ensuring a smooth narrative flow even as the prompt changes from playing with a ball to interacting with a dog.
Conclusion
MemFlow demonstrates that efficient long-form video generation does not require massive memory compression. Instead, by using text-guided sparse retrieval, models can maintain high-fidelity narrative threads across long durations. Future work will explore more sophisticated representative frame selection and multi-character interactions to further push the boundaries of AI-generated cinematography.
Summary updated