TL;DR
vid2vid-long introduces a correspondence-based memory mechanism that enables real-time video-to-video translation with long-range temporal consistency. By using the input video as an explicit index to retrieve previously generated output patches, the model prevents "identity drift" and maintains a consistent world state even when objects leave and re-enter the frame minutes later.
What Problem Does This Paper Address?
Interactive video applications, such as gaming, AR, and VR, require high-quality video translation at real-time speeds. To achieve this, modern autoregressive models typically use a fixed-length sliding window (e.g., only looking at the last 2 seconds of history). While this keeps the computation fast and memory usage constant, it creates a "recency bias."
If an object leaves the frame or is occluded for longer than the window size, the model "forgets" what it looked like. When the object reappears, the model might regenerate it with different colors or textures—a phenomenon known as identity drift. Naively increasing the window size isn't an option, as the memory cost of the Key-Value (KV) cache grows linearly and the attention computation grows quadratically with video length.
Methods
The authors propose vid2vid-long, which moves beyond temporal proximity to find relevant context. Their core insight is that in video-to-video translation, the input video provides a "free" map for the output: if two regions match in the input video, their generated outputs should match as well.
Key components include:
- Compressed Patch History: Instead of storing every frame, the system maintains a dynamic set of input-output patch pairs. The input patch acts as a retrieval key, and the generated output patch acts as the value.
- Input-Driven Retrieval: For every new frame, the model "patchifies" the input and queries the memory. It retrieves the $k$ most similar past output patches based on the similarity of their corresponding inputs.
- Correspondence Cross-Attention: The authors insert new cross-attention layers into the diffusion transformer. These layers allow the current generation process to attend specifically to the retrieved historical patches.
- Novelty-Based Writing: To keep the memory footprint small, the model only writes a new patch to memory if it is "novel" (i.e., its similarity to existing memory is below a threshold $\tau_h$). This allows the memory to scale with the visual complexity of the scene rather than the raw length of the video.
Datasets
The model was evaluated on two primary benchmarks:
- VACE-Benchmark: A collection of diverse real-world scenes. The authors used a synthetic "ABAB" protocol, concatenating two distinct clips in an interleaved order to force the model to revisit previously seen content after a long delay.
- DL3DV: A geometry-grounded 3D dataset. The researchers used calibrated camera poses and 3D Gaussian Splatting to create perfectly looping videos, providing a ground-truth baseline for how a scene should look when the camera returns to a previous viewpoint.
Key Insights and Ideas
- Input as a Global Index: The most powerful idea here is offloading the "discovery" of long-range relationships from the transformer's expensive attention layers to an external index based on the input stream.
- Sub-linear VRAM Scaling: Because the memory only grows when new visual information appears, the VRAM usage remains nearly constant. The authors demonstrate that for a 10-minute video, their method uses only ~0.1 GB of additional VRAM, whereas a naive full-history approach would crash an H100 GPU in under 30 seconds.
- Latent-Space Keys: Interestingly, the authors found that using the VAE's own latent features as retrieval keys worked better and faster than using heavy external vision backbones like DINOv2 or CLIP.
Key Results
The researchers integrated vid2vid-long into several state-of-the-art autoregressive backbones (Self-Forcing, LongLive, and Rolling Forcing).
- Consistency Gains: On the VACE-Bench, the method improved LPIPS (a measure of perceptual reconstruction error) by 37-47% compared to the base models.
- Efficiency: The system maintains interactive speeds, achieving ~13.9 FPS (2-step generation) or ~8.1 FPS (5-step generation) on a single NVIDIA H100.
- Qualitative Quality: As shown in the quantitative evaluation (Table 1), the method achieves the best reconstruction and quality scores among all real-time competitors.
Limitations and Future Work
- Fast Motion: The model struggles with very fast videos where reliable cross-frame correspondences are difficult to establish.
- Conflicting Signals: If a text prompt strongly contradicts the structural input (e.g., a depth map), the model may fail to reconcile the two, leading to artifacts.
- Dynamic Scenes: While the method handles camera movement and reappearing objects well, extremely complex dynamic deformations in the input might still challenge the patch-level matching logic.
Summary updated