Introduction
Generating coherent, long-form video from a first-person (egocentric) perspective remains a significant challenge in computer vision. While current diffusion models excel at short clips, they often suffer from "content drift"—a phenomenon where the visual identity of objects and environments degrades or transforms over time. EgoLCD addresses this by reframing video synthesis as a memory management problem, utilizing a dual-memory architecture to ensure long-term consistency.
The Challenge of Content Drift
In egocentric video, the camera is typically mounted on a user's head, leading to rapid motion and frequent occlusions. Standard autoregressive models predict the next frame based on a limited window of previous frames. As the sequence length increases, the model's "memory" becomes cluttered or fades, leading to "generative forgetting."
A kitchen that starts with blue tiles might gradually morph into a different room, or a specific tool might change shape mid-action. Standard metrics often miss this because they average quality over the whole video, ignoring the fact that the end of the video no longer matches the beginning.
Methodology: The Dual-Memory Architecture
The EgoLCD (Egocentric Long Context Diffusion) framework introduces a "dual-memory" architecture designed to keep the video on track over extended durations. The core insight is that long-form video is not just about predicting the next pixel, but about managing what information to keep and what to discard.
- Long-Term Sparse KV Cache: This serves as the model's "global anchor." It stores historical key-value pairs from previous segments. To prevent computational costs from exploding, it uses a sparsity mechanism that only retains the most "important" tokens based on attention scores. This allows the model to maintain global coherence across long sequences.
- Short-Term Memory with LoRA: Immediate context is handled via a standard attention window enhanced with Low-Rank Adaptation (LoRA). These parameters act as implicit memory units, helping the model quickly adapt to local changes—such as a sudden head turn—without disrupting the underlying scene structure.
- Memory Regulation Loss: This is a crucial training innovation. It forces the model to align its current generation with semantically retrieved "anchors" from its long-term memory. By "tethering" the diffusion process to historical data, the model is prevented from hallucinating new, inconsistent environments.
To guide the generation process, EgoLCD also uses Structured Narrative Prompting (SNP). By leveraging large language models like GPT-4o, the system generates a temporally ordered script of detailed captions, ensuring that actions follow a logical, human-centric sequence.
Evaluation and Results
The researchers evaluated EgoLCD on the EgoVid-5M dataset, a massive-scale benchmark for egocentric activities. To better quantify long-term stability, they introduced the Normalized Referenced Drifting Penalty (NRDP), which specifically measures how much a video's quality and consistency deviate from the initial frame over time.
EgoLCD achieved state-of-the-art results, particularly in consistency metrics. For instance, its Subject Drift score was 0.0844, significantly lower than competing models like MAGI (0.6618) and Framepack (4.3984). Ablation studies confirmed that the Memory Regulation Loss is critical; removing it led to a sharp increase in visual drift, proving that architectural memory requires explicit supervision to function correctly.
Conclusion
By treating long-form video generation as a memory management task rather than a simple sequence prediction, EgoLCD sets a new standard for egocentric synthesis. While challenges remain regarding GPU resource requirements and prompt sensitivity, the use of sparse long-term caches and explicit memory regulation provides a robust path forward for creating stable, minute-long first-person videos.
Summary updated