Modern simulators make video look clean through temporal anti-aliasing: they blend each new frame with a reprojected history of earlier frames. When something moves fast, that history is stale, and the engine leaves a faint translucent trail behind the object. Real cameras do not produce this trail. A real camera smears a moving object while the shutter is open, and the smear is roughly centered on where the object was halfway through the exposure. The ground-truth masks and boxes that simulators export are read from geometry buffers before this post-processing runs, so they mark the object's true current position. A model trained on such data therefore learns from pixels whose trailing edge is ghosted while the label is not. We hypothesize that this teaches a directional bias: synthetic-trained detectors and segmenters will misplace the trailing edges of moving objects on real video, by an amount that grows with image-plane speed.
This note proposes a controlled test of that hypothesis and a cheap fix. The test renders the same scenarios under different temporal pipelines and measures edge errors on real driving video separately for edges that lead and edges that trail the motion. The fix replaces engine ghosting with a physically shaped camera exposure that is consistent with the labels. If the bias exists, it is a sim-to-real gap that no current benchmark isolates. If it does not exist, the community learns that temporal rendering choices can be ignored. Both outcomes are useful, and the experiment is cheap to run.
Core Claim
Our claim is that temporal accumulation in real-time renderers is a separate, measurable axis of the sim-to-real gap. Its signature is predictable: localization error at an object's trailing edge exceeds error at its leading edge, and the gap scales with pixel speed. The signature should vanish when training frames carry symmetric, exposure-style motion blur aligned with the labels.
The prior work that motivates this claim documents the mechanism but stops before its machine-learning consequences. Overview of Real-Time Global Illumination Rendering Methods without Pre-Processing reviews real-time pipelines that render about one sample per pixel and rely on temporal reprojection. It documents that reprojection fails at disocclusions and under fast motion, producing smearing and trails. It does not ask what this does to models trained on the output.
Three dataset papers build on game engines and leave the question untouched:
- Virtual KITTI 2 moves to Unity's high-definition pipeline. It reports that better materials roughly halve the gap, with 91.2% MOTA on synthetic clones against 91.5% on real KITTI. It treats every rendered frame as pristine.
- SHIFT studies continuous weather and lighting shift in CARLA, a driving simulator. It never separates out the renderer's own temporal artifacts.
- Driving in the Matrix harvests stencil and depth buffers from GTA V. That is exactly the setup where labels are taken before post-processing and pixels after, but the paper evaluates single frames and never conditions on motion.
The closest physical-fidelity audit, GAUGE, measures whether simulated trajectories obey mechanics. For example, it finds an apparent gravitational acceleration of 1.84 m/s² in one video model. It does not examine the image-formation stage that turns those trajectories into pixels. Our proposal differs from all of these in what it varies, which is the temporal rendering pipeline with scenes held fixed, and in what it measures, which is error broken down by motion direction rather than aggregate accuracy.
Research Program
The work has three parts.
A diagnostic metric. For each moving object in real test video, we estimate its image-plane velocity from tracked boxes. We split localization error into the box edge or mask contour that faces the motion and the one that faces away. We define trailing-edge asymmetry as the mean signed offset at trailing edges minus that at leading edges, binned by pixel speed. A model with no directional bias shows zero asymmetry at all speeds. Our hypothesis predicts asymmetry that rises with speed for engine-default training and stays flat otherwise. This metric matters because aggregate mAP averages over mostly slow or static objects and can hide the effect entirely.
Controlled rendering arms. CARLA is open source, so we render identical scenario seeds, spawn positions and weather, in the style of SHIFT, under four temporal pipelines:
- Engine default with temporal anti-aliasing on.
- Temporal anti-aliasing off, replaced by purely spatial anti-aliasing.
- Physically simulated exposure: render at several times the target frame rate with temporal accumulation off, then average subframes over a shutter interval. Labels are taken at mid-exposure.
- Flow-based exposure synthesis: blur along the engine's rendered optical flow with a symmetric kernel, also with mid-exposure labels.
Arm 3 is the physically faithful but more expensive reference. Arm 4 is the cheap approximation we would recommend if it matches arm 3.
The key ablation. This arm has the same blur energy as arm 4, but the kernel is one-sided, trailing behind the motion. It also keeps current-frame labels. This mimics the shape of a ghost while controlling for the total amount of blur. If arm 4 beats this arm, direction and label alignment matter, not merely the presence of motion degradation.
Baselines
The naive reference point is the engine-default arm: models trained on CARLA output exactly as shipped, as SHIFT and similar datasets provide it. Any claimed effect is measured against this arm, with identical scenes, frame counts, architecture and schedule.
The strongest simple competitor is the arm with temporal anti-aliasing off. A reviewer would rightly propose it first. CARLA is mostly rasterized, so the one-sample-per-pixel noise argument made for ray-traced pipelines in the global-illumination survey applies only weakly. This arm costs nothing extra to render. We give it the same tuning budget: three learning-rate settings and the same augmentation stack, including standard photometric jitter. Beating it means lower trailing-edge asymmetry and higher real-KITTI accuracy on the high-speed bin, with a confidence interval over three seeds that excludes zero.
The strongest existing domain-adaptation method in the file is pixel-level adversarial translation, Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks. It maps synthetic images toward real appearance one frame at a time and reports 98.2% on MNIST to MNIST-M and a pose error of 23.5° on LineMod. We run it frame-wise on the engine-default renders with unlabeled real KITTI frames as the target, and give it the same compute budget as our arms. Beating it on the high-speed bin while matching it overall would show that a motion-specific fix is complementary to appearance alignment rather than subsumed by it.
As an external anchor, the strongest reported synthetic-to-KITTI vehicle detection numbers in the file are 0.6803 mAP on KITTI Easy and 0.4207 on Hard. Both are for Faster R-CNN trained on Sim 200k in Driving in the Matrix. Our data volume and engine differ, so we report these for context, not as a target we claim to beat. We found no reported result on Lacuna for any sim-to-real comparison with temporal anti-aliasing on versus off.
The ablation with the one-sided kernel and current-frame labels, described above, isolates the proposal's key ingredient.
Why not just…?
Why not just turn temporal anti-aliasing off? This is the real threat, and we treat it as an experimental arm rather than arguing it away. Turning it off removes the ghost but leaves frames that are temporally too sharp. Real cameras blur along motion, so we predict a smaller, opposite-signed residual gap at high speeds. That is a symmetric under-blur error rather than a trailing bias. If the arm with anti-aliasing off matches exposure simulation on both the asymmetry metric and accuracy, our recommendation becomes "disable it." The diagnostic metric would then be the main contribution, and we would say so plainly.
Why not just add generic motion-blur augmentation? Standard augmentation blurs in random directions and leaves labels at the current frame. It neither removes the engine's trailing ghost nor aligns labels with mid-exposure. The one-sided ablation, together with an arm that adds random-direction blur on top of engine-default renders, tests this directly. If random blur closes the asymmetry, flow-aligned exposure is unnecessary.
Why not just use unsupervised domain adaptation? Adversarial alignment can in principle learn any low-level difference. However, Adversarially Masking Synthetic to Mimic Real shows that discriminators latch onto the presence or absence of low-level artifacts, a "shortcut" effect, and that explicitly corrupting synthetic data to mimic sensor defects works better: 22.8% versus 15.2% mIoU on SynLiDAR to SemanticKITTI. Frame-wise translation also cannot move a label that is misaligned with its pixels. The pixel-level baseline will show empirically whether it reduces trailing-edge asymmetry.
Why not just flow-warp predictions at test time? Test-time temporal smoothing addresses flicker in the output. It cannot correct a bias the model learned in training, and warping fails at disocclusions, as Adaptive Exponential Smoothing argues when it motivates avoiding dense flow. We will apply that smoother to every arm's outputs to confirm the training-time ranking holds after smoothing.
Proposed Experiments
We train a Faster R-CNN vehicle detector and a semantic segmenter on each rendering arm, with matched frame counts and three seeds. We evaluate on real KITTI tracking sequences, on the KITTI semantic frames used in Virtual KITTI 2, and on SHIFT's held-out synthetic sequences as a within-domain control.
Primary metrics are mAP and mIoU, stratified by pixel speed, plus trailing-edge asymmetry. The high-speed bin is drawn from turns and oncoming traffic, because typical KITTI motion is slow. We report accuracy against rendering cost for each arm so that the cost of the more expensive exposure simulation is visible.
Three outcomes are decisive:
- Mechanism confirmed. Trailing-edge asymmetry is positive and grows with speed for engine-default training, and is near zero for the exposure arms.
- Cheap fix validated. The flow-based synthesis arm statistically matches the subframe-averaging arm.
- Direction matters. The one-sided ablation reproduces the engine-default asymmetry despite equal blur energy, which ties the effect to ghost shape and label misalignment rather than blur amount.
A null result on the first outcome, meaning no speed-dependent asymmetry even in the fast bin, would falsify the hypothesis and close the question.
References
- Overview of Real-Time Global Illumination Rendering Methods without Pre-Processing
- Virtual KITTI 2
- SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Adaptive Exponential Smoothing for Online Filtering of Pixel Prediction Maps