Consistent Video-to-Video Translation via Explicit Correspondences

Gaurav Parmar, Zhengqi Li, Richard Zhang, Jun-Yan Zhu, Srinivasa Narasimhan, Eli Shechtman, Yotam NitzanarXiv 2026 · arXiv

TL;DR

vid2vid-long introduces a correspondence-based memory mechanism that enables real-time video-to-video translation with long-range temporal consistency. By using the input video as an explicit index to retrieve previously generated output patches, the model prevents "identity drift" and maintains a consistent world state even when objects leave and re-enter the frame minutes later.

What Problem Does This Paper Address?

Interactive video applications, such as gaming, AR, and VR, require high-quality video translation at real-time speeds. To achieve this, modern autoregressive models typically use a fixed-length sliding window (e.g., only looking at the last 2 seconds of history). While this keeps the computation fast and memory usage constant, it creates a "recency bias."

If an object leaves the frame or is occluded for longer than the window size, the model "forgets" what it looked like. When the object reappears, the model might regenerate it with different colors or textures—a phenomenon known as identity drift. Naively increasing the window size isn't an option, as the memory cost of the Key-Value (KV) cache grows linearly and the attention computation grows quadratically with video length.

Fig. 1: Long-range consistency without long context windows. Standard auto- regressive video translation forgets content once it falls outside its context window (middle row). Our proposed method vid2vid-long (bottom row) explicitly retrieves matching past outputs when the inputs recur, maintaining consistency over long videos. Please see the supplement for a large grid of qualitative result videos.

Methods

The authors propose vid2vid-long, which moves beyond temporal proximity to find relevant context. Their core insight is that in video-to-video translation, the input video provides a "free" map for the output: if two regions match in the input video, their generated outputs should match as well.

Key components include:

  • Compressed Patch History: Instead of storing every frame, the system maintains a dynamic set of input-output patch pairs. The input patch acts as a retrieval key, and the generated output patch acts as the value.
  • Input-Driven Retrieval: For every new frame, the model "patchifies" the input and queries the memory. It retrieves the $k$ most similar past output patches based on the similarity of their corresponding inputs.
  • Correspondence Cross-Attention: The authors insert new cross-attention layers into the diffusion transformer. These layers allow the current generation process to attend specifically to the retrieved historical patches.
  • Novelty-Based Writing: To keep the memory footprint small, the model only writes a new patch to memory if it is "novel" (i.e., its similarity to existing memory is below a threshold $\tau_h$). This allows the memory to scale with the visual complexity of the scene rather than the raw length of the video.
Fig. 2: Correspondence-based video translation. The memory is populated with input-output correspondence pairs from previous generations. To support the consis- tent translation of the next block of frames, we first retrieve from memory the output patches whose corresponding input is most similar to the current. These output patches, together with the normal input signal, is the input to our video diffusion model. Based on the new induced input-output pairs, we write to memory only those that are suffi- ciently different from existing.

Datasets

The model was evaluated on two primary benchmarks:

  1. VACE-Benchmark: A collection of diverse real-world scenes. The authors used a synthetic "ABAB" protocol, concatenating two distinct clips in an interleaved order to force the model to revisit previously seen content after a long delay.
  2. DL3DV: A geometry-grounded 3D dataset. The researchers used calibrated camera poses and 3D Gaussian Splatting to create perfectly looping videos, providing a ground-truth baseline for how a scene should look when the camera returns to a previous viewpoint.

Key Insights and Ideas

  • Input as a Global Index: The most powerful idea here is offloading the "discovery" of long-range relationships from the transformer's expensive attention layers to an external index based on the input stream.
  • Sub-linear VRAM Scaling: Because the memory only grows when new visual information appears, the VRAM usage remains nearly constant. The authors demonstrate that for a 10-minute video, their method uses only ~0.1 GB of additional VRAM, whereas a naive full-history approach would crash an H100 GPU in under 30 seconds.
  • Latent-Space Keys: Interestingly, the authors found that using the VAE's own latent features as retrieval keys worked better and faster than using heavy external vision backbones like DINOv2 or CLIP.

Key Results

The researchers integrated vid2vid-long into several state-of-the-art autoregressive backbones (Self-Forcing, LongLive, and Rolling Forcing).

  • Consistency Gains: On the VACE-Bench, the method improved LPIPS (a measure of perceptual reconstruction error) by 37-47% compared to the base models.
  • Efficiency: The system maintains interactive speeds, achieving ~13.9 FPS (2-step generation) or ~8.1 FPS (5-step generation) on a single NVIDIA H100.
  • Qualitative Quality: As shown in the quantitative evaluation (Table 1), the method achieves the best reconstruction and quality scores among all real-time competitors.
Table 1: Quantitative evaluation on depth-to-video translation. We report long-context reconstruction metrics (LPIPS, DreamSim, CLIP-Sim), quality scores (V- Bench), and inference efficiency. Offline methods are shown in gray on the top and distillation based autoregressive methods are shown at the bottom. The best method in each column is marked in bold and the second best method is marked in underline.

Limitations and Future Work

  • Fast Motion: The model struggles with very fast videos where reliable cross-frame correspondences are difficult to establish.
  • Conflicting Signals: If a text prompt strongly contradicts the structural input (e.g., a depth map), the model may fail to reconcile the two, leading to artifacts.
  • Dynamic Scenes: While the method handles camera movement and reappearing objects well, extremely complex dynamic deformations in the input might still challenge the patch-level matching logic.

Summary updated

Abstract

Interactive video-to-video applications require real-time generation while maintaining long-range temporal consistency. However, recent methods achieve speed by restricting attention to a fixed-length causal window. This limits long-range consistency whenever relevant history falls outside the context window. To address this limitation, we introduce vid2vid-long, a correspondence-based approach grounded in a simple observation. Video-to-video translation provides an explicit cross-frame reference: the regions that match in the input video should match in the output video too. Concretely, we compute patch-level correspondences in the input, guiding retrieval of previously generated outputs in the corresponding regions, and condition the model on these patches through correspondence cross-attention. Our experiments demonstrate that vid2vid-long consistently improves long-range temporal consistency across recent video models while preserving generation quality and real-time performance.
BibTeX
@misc{parmar2026consistent,
  author = {Gaurav Parmar and Zhengqi Li and Richard Zhang and Jun-Yan Zhu and Srinivasa Narasimhan and Eli Shechtman and Yotam Nitzan},
  title = {{Consistent Video-to-Video Translation via Explicit Correspondences}},
  year = {2026},
  eprint = {2611.17298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17298},
}
Figure