Generating realistic driving data is a holy grail for autonomous vehicle (AV) development. To safely deploy AVs, we need to stress-test them against a "long tail" of rare conditions—blinding snowstorms, foggy dawns, or torrential rain. While generative AI can "edit" existing driving logs to simulate these conditions, general-purpose video editors often fail the "driving test." They might hallucinate a new traffic light, make a car disappear mid-frame, or warp lane markings, violating the strict Operational Design Domain (ODD) rules that govern real-world traffic.
HorizonComposer is a new framework designed to solve these issues, ensuring that edited driving videos remain spatiotemporally consistent and semantically faithful to the original scene.
TL;DR
HorizonComposer is a framework for instruction-guided driving video editing that preserves critical traffic semantics like lane markings and vehicle identities. By combining a hybrid real-synthetic dataset with dense pixel-level reinforcement learning and "thinking frames," it achieves state-of-the-art consistency and significantly boosts performance for downstream autonomous driving tasks.
What Problem Does This Paper Address?
The core challenge in driving video editing is maintaining traffic semantics. In a typical video of a cat, a slight change in fur texture across frames is a minor artifact. In a driving video, a slight shift in a double-yellow line or a flickering car is a catastrophic failure for an AV's perception system.
Existing models suffer from three main bottlenecks:
- Semantic Corruption: Training on unpaired real-world data (e.g., the same road on different days with different traffic) misleads models into thinking vehicle locations can change arbitrarily during an edit.
- Misaligned Synthetic Priors: Synthetic training pairs often contain "creative additions" that violate road rules.
- Temporal Shock: Injecting a static edited image as a condition for a video often causes "shocks"—unnatural transitions and flickering as the model tries to reconcile the static image with dynamic video context.
Methods
The authors propose a three-pronged approach to inject and protect traffic semantics:
-
Hybrid Video Pairing: They created a massive dataset by pairing real-world videos of the same routes (using pose alignment and temporal filtering) with high-quality synthetic edits. This balances foundational scene structure with clear appearance signals. The dataset construction pipeline combines real-world alignment with synthetic generation.
-
Dense Reward Offline RL: Instead of using a single scalar "score" for a whole video, they use pixel-aligned reward maps. These maps use Segment Anything (SAM) and Vision-Language Models (VLMs) to identify which specific pixels correctly preserve lanes and vehicles. They then use offline reinforcement learning (specifically a dense version of CFGRL) to steer the model toward these reliable regions. Dense reward maps guide the model to preserve content and maintain temporal consistency at the pixel level.
-
Temporal Thinking Frames: To solve "temporal shock," the model doesn't start the video abruptly. It prepends "thinking frames"—temporally reflected versions of the start of the clip—that act as a buffer. This allows the model to "warm up" and smoothly align the new appearance with the rigid geometric constraints of the scene before producing the final output.
Datasets
- Boreas: Used for real-world pose-paired videos across different seasons and weather.
- nuScenes & Waymo: Used to generate the synthetic portion of the training data.
- nuPlan: Used as an out-of-distribution (OOD) test set to evaluate generalizability.
- SeRM & BDD100K: Used to evaluate the impact of the generated data on downstream tasks like lane segmentation and object detection.
- The HorizonComposer Dataset: A newly introduced comprehensive dataset containing 86,614 videos covering 172 categories of driving semantics.
Key Insights and Ideas
- Dense Rewards are Transformative: In complex driving scenes, a training video is rarely 100% "bad." Dense rewards allow the model to extract useful policy improvements from the "good" parts of imperfect data while ignoring regions with hallucinations.
- The Buffer Effect: "Thinking frames" are a simple but effective architectural trick. By giving the diffusion process a temporal "runway," the model can resolve discrepancies between the static reference image and the underlying video dynamics without introducing artifacts into the final result.
- Hybrid Supervision is Necessary: Real data provides the "ground truth" for structure, but synthetic data provides the "clean" signal for appearance changes. Using both is significantly better than using either in isolation.
Key Results
HorizonComposer establishes a new state-of-the-art for driving video editing:
- Content Preservation: Achieved a 39% relative improvement in user preference for content preservation over the next best method.
- Consistency: Improved temporal consistency by 12% in user studies.
- Downstream Utility: Using HorizonComposer-edited videos as data augmentation led to a 5% gain in road line segmentation (mIoU) and a 7% gain in multi-object detection (mAP@75).
- Generalization: It was the only model tested that maintained its performance gains on out-of-distribution data (nuPlan), proving the robustness of its semantic grounding.
Limitations and Future Work
- Reward Model Dependency: The system's performance is capped by the accuracy of the reward evaluators (like SAM and InternVL). If the reward model misses a small, distant agent, the generator might still corrupt it.
- Global vs. Local: While the model excels at global changes (weather/time), it is not a standalone "local" editor (e.g., "add a truck here"). However, the authors showed it can be successfully composed with other tools like HorizonForge for local edits.
- Future Directions: The authors suggest exploring joint training of the reward and diffusion models to further reduce noise in the supervision signal.
Summary updated