HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics

Mauricio Soroco, Yuqiu Liu, Zaid Tasneem, Francesco Pittaluga, Abhishek Aich, Wuyang Chen, Manmohan Chandraker, Ziyu JiangarXiv 2026 · arXiv

Generating realistic driving data is a holy grail for autonomous vehicle (AV) development. To safely deploy AVs, we need to stress-test them against a "long tail" of rare conditions—blinding snowstorms, foggy dawns, or torrential rain. While generative AI can "edit" existing driving logs to simulate these conditions, general-purpose video editors often fail the "driving test." They might hallucinate a new traffic light, make a car disappear mid-frame, or warp lane markings, violating the strict Operational Design Domain (ODD) rules that govern real-world traffic.

HorizonComposer is a new framework designed to solve these issues, ensuring that edited driving videos remain spatiotemporally consistent and semantically faithful to the original scene.

HorizonComposer maintains complex traffic semantics across various global and local edits.

TL;DR

HorizonComposer is a framework for instruction-guided driving video editing that preserves critical traffic semantics like lane markings and vehicle identities. By combining a hybrid real-synthetic dataset with dense pixel-level reinforcement learning and "thinking frames," it achieves state-of-the-art consistency and significantly boosts performance for downstream autonomous driving tasks.

What Problem Does This Paper Address?

The core challenge in driving video editing is maintaining traffic semantics. In a typical video of a cat, a slight change in fur texture across frames is a minor artifact. In a driving video, a slight shift in a double-yellow line or a flickering car is a catastrophic failure for an AV's perception system.

Existing models suffer from three main bottlenecks:

  1. Semantic Corruption: Training on unpaired real-world data (e.g., the same road on different days with different traffic) misleads models into thinking vehicle locations can change arbitrarily during an edit.
  2. Misaligned Synthetic Priors: Synthetic training pairs often contain "creative additions" that violate road rules.
  3. Temporal Shock: Injecting a static edited image as a condition for a video often causes "shocks"—unnatural transitions and flickering as the model tries to reconcile the static image with dynamic video context.

Methods

The authors propose a three-pronged approach to inject and protect traffic semantics:

  1. Hybrid Video Pairing: They created a massive dataset by pairing real-world videos of the same routes (using pose alignment and temporal filtering) with high-quality synthetic edits. This balances foundational scene structure with clear appearance signals. The dataset construction pipeline combines real-world alignment with synthetic generation.

  2. Dense Reward Offline RL: Instead of using a single scalar "score" for a whole video, they use pixel-aligned reward maps. These maps use Segment Anything (SAM) and Vision-Language Models (VLMs) to identify which specific pixels correctly preserve lanes and vehicles. They then use offline reinforcement learning (specifically a dense version of CFGRL) to steer the model toward these reliable regions. Dense reward maps guide the model to preserve content and maintain temporal consistency at the pixel level.

  3. Temporal Thinking Frames: To solve "temporal shock," the model doesn't start the video abruptly. It prepends "thinking frames"—temporally reflected versions of the start of the clip—that act as a buffer. This allows the model to "warm up" and smoothly align the new appearance with the rigid geometric constraints of the scene before producing the final output.

Figure 2: Dataset Construction and Pseudo-Labeling Pipeline. (Top) Realistic Dataset Devel- opment : Real-world data (Sec. 3.2.1) are matched via pose alignment, temporally filtered, and processed to generate editing instructions, forming real-world video pairs. (Bottom) Pseudo Dataset Development (Sec. 3.2.2): We synthesize driving scenes by editing the initial frame with an image prior, followed by a video generative model to propagate the edits and produce synthetic video pairs.
Dense reward maps guide the model to preserve content and maintain temporal consistency at the pixel level.

Datasets

  • Boreas: Used for real-world pose-paired videos across different seasons and weather.
  • nuScenes & Waymo: Used to generate the synthetic portion of the training data.
  • nuPlan: Used as an out-of-distribution (OOD) test set to evaluate generalizability.
  • SeRM & BDD100K: Used to evaluate the impact of the generated data on downstream tasks like lane segmentation and object detection.
  • The HorizonComposer Dataset: A newly introduced comprehensive dataset containing 86,614 videos covering 172 categories of driving semantics.

Key Insights and Ideas

  • Dense Rewards are Transformative: In complex driving scenes, a training video is rarely 100% "bad." Dense rewards allow the model to extract useful policy improvements from the "good" parts of imperfect data while ignoring regions with hallucinations.
  • The Buffer Effect: "Thinking frames" are a simple but effective architectural trick. By giving the diffusion process a temporal "runway," the model can resolve discrepancies between the static reference image and the underlying video dynamics without introducing artifacts into the final result.
  • Hybrid Supervision is Necessary: Real data provides the "ground truth" for structure, but synthetic data provides the "clean" signal for appearance changes. Using both is significantly better than using either in isolation.

Key Results

HorizonComposer establishes a new state-of-the-art for driving video editing:

  • Content Preservation: Achieved a 39% relative improvement in user preference for content preservation over the next best method.
  • Consistency: Improved temporal consistency by 12% in user studies.
  • Downstream Utility: Using HorizonComposer-edited videos as data augmentation led to a 5% gain in road line segmentation (mIoU) and a 7% gain in multi-object detection (mAP@75).
  • Generalization: It was the only model tested that maintained its performance gains on out-of-distribution data (nuPlan), proving the robustness of its semantic grounding.
Figure 4: Qualitative Comparison of Driving Scene Editing. We highlight key differences in semantic preservation and visual quality across methods. Target conditions are shown in text. Row order from top to bottom: input video, CCEdit, UniVideo, SyntheticPip, Ours. (a) SyntheticPip hallucinates a yellow traffic light from the initial image condition. (b) SyntheticPip introduces a new vehicle that suddenly disappears, breaking temporal consistency. (c) Only our method and UniVideo accurately retain the “STOP AHEAD” road marking. Overall, CCEdit regularly suffers from hazy artifacts and UniVideo distorts visual realism, whereas our approach maintains high-fidelity instruction alignment and strict content preservation.

Limitations and Future Work

  • Reward Model Dependency: The system's performance is capped by the accuracy of the reward evaluators (like SAM and InternVL). If the reward model misses a small, distant agent, the generator might still corrupt it.
  • Global vs. Local: While the model excels at global changes (weather/time), it is not a standalone "local" editor (e.g., "add a truck here"). However, the authors showed it can be successfully composed with other tools like HorizonForge for local edits.
  • Future Directions: The authors suggest exploring joint training of the reward and diffusion models to further reduce noise in the supervision signal.

Summary updated

Abstract

Instruction-guided video generative models offer a scalable solution for stress-testing autonomous driving systems by simulating diverse environmental conditions. However, driving scenes operate within a dense Operational Design Domain (ODD) governed by strict traffic semantics. When applied to global driving-condition edits, general-purpose instruction-guided video editors frequently corrupt these semantics, leading to structural hallucinations and ``temporal shock.'' This work distills and protects traffic semantics during video editing at three levels. First, we construct a hybrid dataset by combining temporally aligned real-world videos with synthetic edits, balancing foundational scene structure with unambiguous appearance signals. Second, during training, we adapt advantage conditioning from offline reinforcement learning to align the model using dense rewards. We extract dense, pixel-level rewards from the hybrid data to serve as a localized signal and actively steer generation toward structurally reliable pixels. Finally, our model mitigates temporal shock by introducing thinking frames, a transition mechanism that seamlessly reconciles static image conditioning with dynamic video context. Extensive experiments demonstrate that our method prevents the corruption of multi-actor dynamics, establishing state-of-the-art spatiotemporal consistency, yielding consistent user preference (39% relative improvement in content preservation and 12% in temporal consistency), and achieving a significant 5% gain in downstream road line segmentation and 7% gain in downstream multi-object detection.
BibTeX
@misc{soroco2026horizoncomposer,
  author = {Mauricio Soroco and Yuqiu Liu and Zaid Tasneem and Francesco Pittaluga and Abhishek Aich and Wuyang Chen and Manmohan Chandraker and Ziyu Jiang},
  title = {{HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics}},
  year = {2026},
  eprint = {2610.28829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.28829},
}
Figure