Training Drone Trajectory Predictors on What the Detector Sees

A trajectory predictor on a real tracking system never sees the true path of its target. It sees what an object detector reports: box centers that jitter, vanish for a few frames when the target blurs or blends into the background, and arrive tens of milliseconds late. Synthetic training data for agile targets such as small drones currently ignores this. The strongest example generates smooth, physically plausible quadrotor paths and projects them into clean pixel coordinates. The result beats a Kalman filter on real footage, but the model is trained on perfect observations and deployed on imperfect ones.

We propose that, for agile targets, the part of the synthetic-to-real gap that matters most for trajectory prediction is the gap in the observation process. That gap is structured: detector failures depend on the motion itself, because fast turns produce motion blur and blur produces missed or displaced detections. We will build a generator that renders a scanned drone along physically generated flight paths, composites it with exposure-accurate motion blur onto real surveillance backgrounds, runs the real deployed detector on the result, and injects the measured pipeline latency. The predictor is then trained on detector outputs rather than ground truth. The main question is whether this beats much cheaper ways of corrupting clean synthetic paths, and where it does.

Core Claim

Our claim is that a forecaster trained on rendered-then-detected synthetic tracks will beat one trained on clean or noise-corrupted synthetic coordinates, when both are evaluated on real ANTI-UAV footage with a real detector in the loop. We further claim the advantage will be concentrated in high-speed, high-curvature segments, where detector errors are most strongly tied to motion. If the gain is uniform across motion regimes, the claim fails. So does the whole proposal if a motion-conditioned noise model fitted to real detector residuals matches the rendered pipeline. In that case, rendering is unnecessary.

What Prior Work Does and Leaves Open

The motivating result is Generating Synthetic Training Data for Deep Learning-Based UAV Trajectory Prediction. It synthesizes 1,000 minimum-snap trajectories, which are paths that minimize the fourth derivative of position and so resemble aggressive but feasible quadrotor flight. It projects them through randomly placed virtual cameras and trains a recurrent network with a mixture-density output head (RNN-MDN). On ANTI-UAV this model reaches a 12-frame final displacement error (FDE, the pixel distance between predicted and true position at the last forecast step) of 121.45 px on electro-optical 1920×1080 video, against 179.46 px for a constant-velocity Kalman filter. On infrared 640×512 video it reaches 61.46 px against 68.01 px. That work supplies our trajectory generator and our strongest baseline. It produces no images and assumes clean box centers, and that is exactly the gap we target.

Two lines of work shape how we render. Real2Render2Real scans objects with a phone into 3D Gaussian Splatting, a scene representation made of many small colored translucent ellipsoids that rasterize quickly. It then renders randomized robot demonstrations and matches 150 real teleoperated demonstrations. We borrow its scan-and-render recipe. We depart from it in two ways: it is kinematic and quasi-static, while our targets move fast, and it discards the splats for meshes, while we keep them. Hardware-Rasterized Ray-Based Gaussian Splatting reports 170–350 FPS for artifact-reduced splat rendering on a consumer GPU. This is what makes our blur model affordable, because realistic exposure blur requires rendering many sub-frames per output frame.

This is also where rendering latency enters the proposal. It enters as a budget on data generation and as a quantity the predictor must forecast across at deployment. The file does not support a claim that dynamic neural rendering belongs inside the control loop: the dynamic splatting method in 3D Gaussian Representations with Motion Trajectory Field for Dynamic Scene Reconstruction runs at about 26 FPS. We therefore render only a rigid drone asset moved along the trajectory, which the static renderer handles.

Three papers constrain the design. S2R-Bench shows that hand-designed sensor corruptions can badly misestimate real ones. It reports real LiDAR under glare performing 85% worse than simulation predicted. This is the reason we distrust parametric noise and run the real detector instead. Anti-UAV: A Large Multi-Modal Benchmark for UAV Tracking documents thermal crossover, where the drone's heat matches the background. That kind of failure depends on the background, which argues for compositing onto real backgrounds rather than synthetic ones. CUQDS shows that trajectory uncertainty collapses under distribution shift: standard conformal coverage on LaneGCN falls to 0.402, and its online controller restores it to 0.746. We use its method as a calibration check. Learned motion models for multi-object tracking, such as MambaTrack+ in Exploring Learning-based Motion Models in Multi-Object Tracking and KalmanFormer, are trained on real detector boxes from large annotated sets. They supply alternative predictor architectures but not a synthetic data strategy.

Research Program

We first characterize the real observation process. We train a drone detector on the ANTI-UAV training split and run it on held-out training sequences. For each frame we record whether the drone was detected, how far the box center is from the annotation, and the target's image-plane speed and curvature computed from annotations. We also time the detector and tracker on fixed hardware to obtain a latency distribution. This step yields a measured curve of miss rate and center error as a function of motion, and it serves as the reference that any synthetic pipeline must reproduce.

Next we build the generator. We scan two or three quadrotors into Gaussian-splat assets. We generate minimum-snap paths as in Becker et al. and place them in front of real background frames taken from ANTI-UAV training sequences, with the camera models inferred per sequence. Each output frame averages N sub-frame renders spanning the exposure interval, with sensor noise and compression added afterward. For infrared we render intensity from a per-asset temperature contrast sampled relative to the background. We treat this as an approximation and report electro-optical and infrared results separately.

We then run the same real-trained detector on the rendered frames. The resulting box sequences, including misses, are delayed by latencies sampled from the measured distribution. The predictor, first the RNN-MDN from Becker et al. and then a Mamba-style state-space model, is trained to output the target's true position at the future time when its output will actually be used, not at the time of the last observation.

Finally we test the mechanism. Before any forecasting is trained, we compare the synthetic miss-rate-versus-speed curve against the real one from the first step. If rendering does not reproduce the motion-dependence of real detector failures, we will know before spending effort on forecasting, and the proposal is falsified at low cost.

Baselines

The naive reference is a constant-velocity Kalman filter run on the same detector outputs with the same latency compensation. Its reported ANTI-UAV 12-frame FDE is 179.46 px on electro-optical and 68.01 px on infrared, from Generating Synthetic Training Data for Deep Learning-Based UAV Trajectory Prediction. We will re-run it under our protocol rather than rely on that number, because that number was obtained from clean annotated boxes.

The strongest existing method is the clean-synthetic RNN-MDN from the same paper, with reported 12-frame FDE of 121.45 px on electro-optical and 61.46 px on infrared. We retrain it on the identical 1,000-trajectory budget and identical architecture. We evaluate it under two protocols: annotated-box input, to reproduce the reported numbers, and detector-box input, which is our main test. Beating it means a lower 12-frame FDE under detector input, with confidence intervals from bootstrapping over sequences that do not overlap. It also means no regression under annotated-box input.

We add a real-data baseline: the same predictor trained on detector outputs from ANTI-UAV training sequences, which is data a practitioner already has. We also train on real plus rendered-synthetic data. For learned multi-object motion models we found no reported result on Lacuna for ANTI-UAV, so MambaTrack+ appears only as an alternative predictor backbone.

Two ablations isolate the key ingredient. The first trains on clean synthetic paths corrupted by independent Gaussian jitter and random dropouts, matched to the real detector's overall miss rate and error variance. The second uses a motion-conditioned noise model whose miss probability and center error are fitted, as functions of speed and curvature, to the real residuals from the first step. Further ablations remove blur (N=1 sub-frame), remove latency injection, and swap the splat asset for a textured mesh. Every variant receives the same number of trajectories, the same training steps, and the same hyperparameter search of twenty trials on a shared validation split.

Why not just…?

Why not just add noise to Becker et al.'s clean coordinates? Noise matched to the average error rate is cheap and may capture much of the gain. Our claim is that real failures cluster during fast maneuvers, and independent noise cannot place misses where they occur. The independent-noise ablation tests this directly. We will report the result by motion stratum, because an average-only comparison could hide the effect we predict.

Why not just fit a noise model to real detector errors and skip rendering? This is the most serious threat, and it may win. A motion-conditioned noise model fitted on real residuals uses real statistics and costs almost nothing. Rendering can only beat it through effects that speed and curvature do not capture: background-dependent failures such as thermal crossover or clutter, and error correlations across frames. If the fitted model matches rendering within confidence intervals, we will conclude that the observation gap is real but that rendering is not needed to close it. That is still a useful and publishable answer.

Why not just train on real ANTI-UAV detector tracks? Real data is available, with over 580,000 annotated boxes according to Anti-UAV: A Large Multi-Modal Benchmark for UAV Tracking. However, the maneuvers in it are whatever the recorded pilots happened to fly. Synthetic data can cover aggressive paths that are rare in the recordings. The real-only and real-plus-synthetic comparisons, stratified by curvature, will show whether synthetic data adds anything beyond volume. If real-only training wins in every stratum, the synthetic program has no purpose on this benchmark.

Why not just use an extended or unscented Kalman filter tuned for latency? These filters are cheap and principled. However, the Hy-Tracker papers (Hy-Tracker) report that linear-Gaussian motion assumptions lose lock during erratic maneuvers. We include a latency-compensated Kalman filter in every comparison so that the size of this gap is measured under our protocol rather than assumed.

Proposed Experiments and Decisive Checks

All experiments use the ANTI-UAV test sequences and report FDE and average displacement error at horizons of 8, 10 and 12 frames, separately for electro-optical and infrared. A third metric, latency-compensated FDE, measures error at the moment the prediction is consumed. As a downstream measure, we report tracker precision and success when the predictor sets the detector's search window. For calibration, we measure coverage of the mixture-density prediction regions before and after a CUQDS-style online conformal controller.

The first decisive check is a crossover. Under annotated-box input, all synthetic variants should perform about equally, since the observations are clean. Under detector input, the rendered pipeline should beat both the clean and independent-noise variants. If the rendered pipeline helps equally under annotated-box input, its gain comes from something other than the observation process, and our explanation is wrong.

The second check is stratification by target speed and curvature. The advantage of rendering over the fitted noise model should grow with speed, and the synthetic miss-rate curve should track the real one.

The third check sweeps injected latency from zero up to the measured pipeline latency. It asks whether latency-aware training closes most of the gap between FDE at observation time and FDE at use time. The comparison is against applying latency compensation only at test time.

The rendered pipeline is worth its cost only if it wins the first two checks against the fitted noise model. We will state that result plainly whichever way it falls.