LongSCOP: Semantically Consistent Long Video Outpainting

Yu Tang, Zhiyuan You, Zhixiao Wang, Fanghua Yu, Qingyu Zhang, Xiang Yin, Chao Dong, Jinjin GuarXiv 2026 · arXiv

TL;DR

LongSCOP introduces a robust framework for long-horizon video outpainting, overcoming the common pitfalls of "generation collapse" and "semantic drift." By combining a specialized training recipe—including self-forcing and latent calibration—with a VLM-based agent that retrieves informative reference frames from across the entire video, the system maintains visual stability and subject identity for sequences 20 seconds or longer.

What Problem Does This Paper Address?

Video outpainting is the process of expanding the spatial field of view of a video beyond its original boundaries. While recent diffusion models excel at short clips, extending this to long videos (e.g., 20+ seconds) is notoriously difficult.

Most systems use an "iterative" approach, generating the video chunk by chunk. This leads to two primary failure modes:

  1. Generation Collapse: Errors accumulate between chunks, leading to "color spill," flickering, or the generation of abstract, nonsensical blobs.
  2. Semantic Drift: Because the model only sees the immediate past, subjects gradually morph into different entities over time—for example, a human hand slowly transforming into a lizard's body as the video progresses.
Illustration of generation collapse and semantic drift

Methods

LongSCOP (Semantically Consistent long Video Out-Painting) tackles these issues through a foundation of stability and an advanced layer of semantic guidance.

1. The Stability Foundation

To prevent the model from "breaking" over time, the authors introduce three training and inference techniques:

  • Self-forcing Training: During training, the model uses its own previous (potentially noisy) predictions as references rather than ground-truth frames. This aligns the training environment with the reality of inference-time error accumulation.
  • Latent Calibration: The authors found that repeated encoding/decoding cycles distort latent statistics. They introduce a calibration step that aligns the predicted latents with the statistics of the original "real" video frames, preventing brightness and saturation drift.
  • Bidirectional Reference: Instead of looking only at the past, the model looks slightly into the future of the input video to anticipate scene transitions and maintain boundary consistency.
The self-forcing training strategy

2. The VLM-based Retrieval Agent

To solve semantic drift, the authors recognize that a long video often contains the "answers" to its own missing regions in distant frames. They developed a two-stage retrieval agent:

  • Geometry Filtering: Uses the Mast3R model to find candidate frames that have a moderate spatial overlap with the current chunk, ensuring they can be geometrically aligned.
  • Semantic Selection: A Vision Language Model (Qwen3.5-VL) analyzes the remaining candidates to pick the most "informative" frames—those that actually show the subjects or backgrounds currently being outpainted.
The key reference frame retrieval framework

Datasets

The researchers curated a high-quality training dataset of 62,000 video clips covering diverse categories like portraits, architecture, and natural scenes.

For evaluation, they introduced LVO-Bench (Long Video Out-painting Benchmark). Curated by experts, it consists of 52 challenging cases ranging from 5 to 20+ seconds, categorized by environment (indoor/outdoor) and shot scale (close-up to long-shot).

Key Insights and Ideas

  • The "Aha" Moment for References: The authors proved that adding just one well-chosen reference frame from elsewhere in the video can improve outpainting performance by nearly 70 FVD points.
  • VLMs as Reasoning Agents: Traditional metrics like CLIP scores or feature matching often fail to understand what is missing in an outpainted region. By using a VLM, the system can "reason" that if the left side of a frame shows half a person, it should look for a reference frame that shows the rest of that specific person.
  • Training vs. Plug-in: While the retrieval agent can be used as a test-time "plug-in," the authors found that "joint training"—exposing the model to these retrieved references during the fine-tuning phase—significantly reduces the gap between training and inference.

Key Results

LongSCOP significantly outperforms existing state-of-the-art models like M3DDM and VACE, especially on the long-horizon LVO-Bench:

  • FVD (Fréchet Video Distance): Improved by 28.19 points over the previous best model (VACE), indicating much higher visual quality and temporal consistency.
  • PSNR: Achieved a gain of 0.92 dB, showing better pixel-level reconstruction.
  • Qualitative Superiority: As seen in Figure 5, LongSCOP maintains the identity of characters (like the specific clothing of people in a crowd) over 22 seconds, whereas baseline models lose track of these details almost immediately.
Qualitative comparison of LongSCOP against baselines

Limitations and Future Work

Despite its success, LongSCOP has a few hurdles:

  • Computational Overhead: The full pipeline (including VLM retrieval) adds about 49% more inference time compared to the base VACE model.
  • Reference Availability: The semantic consistency module relies on the existence of informative frames within the source video. If a subject appears for the first time and is never seen again, the retrieval agent has nothing to find.
  • Speed: Future work will focus on optimizing the retrieval speed and the efficiency of the self-forcing training process.

Summary updated

Abstract

Video out-painting is an important problem, yet existing research focuses primarily on short clips, leaving long video out-painting largely underexplored. Extending video out-painting to long videos poses substantial challenges. A natural solution is iterative out-painting, i.e., generating videos chunk by chunk, but this introduces two major issues: generation instability or even failure caused by inter-chunk inconsistencies and error accumulation, and semantic drift over long temporal horizons. In this paper, we propose LongSCOP, short for Semantically Consistent long Video Out-Painting. First, we introduce a training recipe with three components for smooth and stable long-horizon generation: self-forcing training, latent calibration, and future references. Beyond generation stability, we recognize that the rich temporal context of long videos provides valuable references for maintaining semantic consistency. We therefore develop a VLM-based agent that automatically selects the most informative reference frames from the entire video to maintain consistent subjects, scenes, and backgrounds throughout long-horizon generation. We also construct a high-quality training dataset of 62K video clips. To facilitate research on this underexplored problem, we introduce LVO-Bench, a benchmark curated by human experts to assess long-horizon stability and semantic consistency. Extensive experiments demonstrate that our method outperforms existing approaches in both generation stability and semantic coherence.
BibTeX
@misc{tang2026longscop,
  author = {Yu Tang and Zhiyuan You and Zhixiao Wang and Fanghua Yu and Qingyu Zhang and Xiang Yin and Chao Dong and Jinjin Gu},
  title = {{LongSCOP: Semantically Consistent Long Video Outpainting}},
  year = {2026},
  eprint = {2611.12445},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12445},
}
Figure