Synthetic video from game engines is now good enough to replace most real training data for tracking people across frames. Chang et al. report that about 30 synthetic frames are worth one real frame, and that 60–80% of real data can be swapped out without loss. There is a catch. Those exchange rates were all measured on clean benchmark footage. Separately, models trained on diffusion-generated images turn out to be markedly more fragile to image corruptions than models trained on real photos. The likely reason is that generators produce images without sensor grain, compression artifacts or blur. If that "too clean" effect also holds for rendered video, then a synthetic frame is worth less on a cheap, noisy, low-light camera than on the benchmark. A practitioner who budgets real-data collection from a clean-set exchange rate would then be systematically misled.
This note proposes to measure the synthetic-to-real exchange rate as a function of how degraded the deployment camera is, instead of as a single number. It then tests which repair restores it. The candidates are generic noise augmentation, rendering the synthetic video through a camera noise model fitted to real sensors, or placing a learned denoiser in front of the tracker. The core claim is falsifiable. Clean-set substitutability overstates the value of synthetic data under realistic sensor noise, and the gap comes specifically from missing sensor statistics in the synthetic set. The gap should therefore close when synthetic frames carry calibrated real-sensor noise, but not when they carry generic Gaussian or algorithmic corruptions.
Core Claim and What Is New
Three lines of prior work motivate the question. None of them answers it.
On the Equivalency, Substitutability, and Flexibility of Synthetic Data supplies the method: substitutability curves and a frame equivalency ratio for multi-person tracking on MOT17 and DanceTrack, using the Unity-based M³Act generator. Every one of its measurements uses clean test video.
Is Synthetic Data all We Need? supplies the motivating evidence. Diffusion-trained "synthetic clones" have roughly 15–20% lower relative accuracy on ImageNet-C than real-trained models. That result is limited to classification, uses algorithmic corruptions rather than physical sensor noise, and says nothing about how much real data one still needs.
A third group constrains the design. Robustness gained from synthetic corruptions tends not to transfer to real ones:
- In Measuring Robustness to Natural Distribution Shifts in Image Classification, models hardened against synthetic corruptions show near-zero effective robustness on natural shifts.
- In Synthetic and Natural Noise Both Break Neural Machine Translation, training on synthetic character swaps fails on real human typos.
- In AlleNoise, methods tuned for synthetic label flips give almost nothing on real label noise.
These results are why the proposal must test on noise derived from real sensors, and why generic augmentation is the hypothesis to beat rather than the solution to assume.
The new object is the noise-conditional exchange rate. Let $H(\cdot)$ be tracking accuracy on a test set degraded at severity $s$. Let $N_{\text{real}}$ be the number of real frames used in fine-tuning, and $N_{\text{syn}}$ the number of synthetic pretraining frames. The exchange rate is defined as
$$r(s) = \frac{N_{\text{syn}}}{\Delta N_{\text{real}}} \quad \text{such that } H_s(N_{\text{syn}}, N_{\text{real}}) = H_s(0, N_{\text{real}} + \Delta N_{\text{real}}),$$
that is, the number of synthetic frames needed to match the accuracy gained from $\Delta N_{\text{real}}$ additional real frames at the same noise severity. At $s = 0$ this should reproduce the reported value of about 30:1. The hypothesis is that $r(s)$ rises steeply with $s$ for plain synthetic data, and stays near flat when synthetic frames are rendered through a calibrated sensor model.
Research Program
Building a realistic deployment noise. Real noise in camera footage depends on signal level, varies across the image, and is shaped by the camera's image-processing pipeline. It is not additive Gaussian noise. Transfer Learning From Synthetic to Real-Noise Denoising With Adaptive Instance Normalization (AINDNet) shows this matters and gives a recipe: pretrain on synthetic images passed through a camera noise model, then adapt on a few real pairs.
We will fit per-camera, signal-dependent noise models to the real noisy/clean pairs in SIDD, the smartphone image denoising dataset. MOT17 and DanceTrack frames will be mapped approximately back to raw sensor space, the fitted noise will be injected, and the frames will be re-processed, over a sweep of light levels.
Tuning on the test noise must be ruled out, so the smartphone models will be split into disjoint groups. Some are used for training-time noise, and others are held out for deployment only. The Darmstadt Noise Dataset (DND) serves as a fully held-out camera family where its data allow noise fitting. For contrast, we also evaluate on ImageNet-C–style corruptions from Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, applied per frame.
One limitation is stated up front. SIDD noise is per-image, so the injected noise has no realistic correlation from one frame to the next. We treat it as independent per frame, and flag the temporal version as future work.
Measuring substitutability curves. We follow the protocol of Chang et al.: synthetic pretraining on M³Act, then fine-tuning on 0, 20, 40, 60, 80 and 100% of real training data, with the MOTRv2 tracker on DanceTrack and a standard detector-plus-association tracker on MOT17. Each curve is traced at every noise severity. Detection accuracy on individual frames is reported alongside HOTA, a tracking score combining detection and identity accuracy. This separates two failure sources: failing to find people in noisy frames, and failing to keep their identities linked over time.
Testing four training conditions on the synthetic side. All four keep the real fine-tuning data fixed:
- plain M³Act renders;
- renders with additive Gaussian noise and ImageNet-C–style augmentation;
- renders passed through the calibrated noise models from the training-side phones;
- plain renders with an auxiliary head that predicts the injected noise and is discarded at test time, modeled on Enhancing Trajectory Prediction through Self-Supervised Waypoint Distortion Prediction.
The fourth condition tests whether learning to separate noise adds anything beyond simply seeing realistic noise.
Baselines
The naive reference is a tracker trained on 100% real, clean data with no synthetic pretraining and no noise handling, evaluated at every severity. Its degradation curve is the floor that every other method must exceed. A second naive point trains the same tracker on 100% real data with calibrated noise augmentation. This tells us whether synthetic data helps at all once real data is noise-augmented.
The strongest existing method is the Chang et al. recipe itself: M³Act pretraining followed by partial real fine-tuning. Chang et al. report that multi-group procedural synthetic data raises MOTRv2 HOTA on DanceTrack from 68.8 to 73.5. They also report that fine-tuning on 20% of real MOT17 data nearly matches 100% real data, but the file gives no numeric HOTA for that MOT17 result. We will first reproduce the 73.5 on clean DanceTrack under identical compute. The comparison is only meaningful if that reproduction lands within noise of the reported value.
Every arm gets the same real data splits, the same number of pretraining steps and GPU budget, and the same hyperparameter search budget, tuned on a clean validation split so that no arm sees held-out camera noise. Beating this baseline means two things:
- at the held-out camera noise, a higher HOTA than the Chang recipe at the same real-data fraction, across three seeds with bootstrap intervals that exclude zero difference;
- a flatter $r(s)$ curve.
The key ablation compares generic-noise synthetic data against calibrated-noise synthetic data at matched perceived noise strength, measured by the peak signal-to-noise ratio (PSNR) of the injected noise. This isolates whether the statistical form of the noise, not its amount, restores substitutability.
Why not just…?
Why not just add Gaussian noise or ImageNet-C augmentations to the synthetic frames? This is the cheapest fix and is included as an arm. The prior evidence goes against it: synthetic-corruption robustness failed to transfer to natural shifts in Taori et al., and synthetic noise failed to transfer to real noise in Belinkov and Bisk and in AlleNoise. Low-level sensor grain is closer to Gaussian noise than typos or natural shifts are, though, so this alternative could genuinely work here. If the generic-noise arm matches the calibrated arm on held-out cameras, we will report that calibration is unnecessary for this task. That is still a useful practical finding, and the noise-conditional exchange rate remains the contribution.
Why not just put a pretrained real-noise denoiser in front of the tracker? This is the most serious threat. AINDNet reaches strong real-noise denoising from only 32 real pairs, so a front-end denoiser might restore clean-like inputs and make the synthetic data's noise content irrelevant. We therefore add an arm with an AINDNet-style denoiser, adapted on the training-side phones, placed in front of the plain-synthetic tracker. It is scored on accuracy and on added inference cost. Denoisers can smooth away small or distant people, which the per-frame detection breakdown will reveal. If the denoiser arm wins everywhere, the conclusion becomes "fix the input, not the training data," and we will say so.
Why not just collect more real noisy footage? That is exactly the cost the exchange rate prices. Chang et al. show dense tracking annotation is expensive enough that synthetic substitution pays. Taori et al. show that even very large real datasets buy only about 2.1% effective robustness. The 100%-real-with-noise baseline measures directly how far real data alone gets.
Why not just use the classification results of Singh et al.? Their numbers come from classification under algorithmic corruptions. They say nothing about how much real data is still needed, or about keeping identities linked over time. Our frame-versus-track decomposition tests whether the "clean data trap" is mostly a detection problem or also an association problem.
Decisive Checks
The proposal succeeds as science if one of three outcomes holds clearly:
- $r(s)$ for plain synthetic data grows substantially from its clean value of about 30:1 at realistic low-light severities on held-out cameras, while the calibrated arm stays near its clean value. This supports the core claim.
- $r(s)$ stays flat for plain synthetic data. This refutes the clean-data-trap extension to rendered video and reassures practitioners.
- The generic-noise or denoiser arm matches the calibrated arm, which settles the main alternatives.
A secondary check reuses the idea behind Is this Model for Real?, which asks whether real and simulated data land in the same regions of a model's parameter space. Here we compare the drop in accuracy between real noisy footage and synthetic footage rendered with the same noise. The question is whether that difference is no larger than the difference between two independent noise draws. If so, the calibrated noise is behaviorally indistinguishable from the real noise.
References
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images
- Measuring Robustness to Natural Distribution Shifts in Image Classification
- Synthetic and Natural Noise Both Break Neural Machine Translation
- AlleNoise - large-scale text classification benchmark dataset with real-world label noise
- Transfer Learning From Synthetic to Real-Noise Denoising With Adaptive Instance Normalization
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- Enhancing Trajectory Prediction through Self-Supervised Waypoint Distortion Prediction
- Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality