A drone that follows a moving target has to keep that target in view. To plan with gradients, the best current methods replace the hard "in view / out of view" test with a smooth surrogate: a soft score that falls off near the edge of the camera's viewing cone. This surrogate is built from geometry alone. It assumes that anything inside the cone gets detected. Real detectors fail inside the cone all the time. They miss targets that are far away, motion-blurred during fast turns, partly hidden behind trees, viewed at steep angles, or seen at night or in fog. A planner trained in simulation against the geometric cone therefore expects detections in places where the real detector stays silent. We propose to close the sim-to-real gap at this level. We would fit a small, smooth model of the probability that the detector fires, as a function of target and camera state, separately on simulated and on real detector logs. We would then train the tracking policy in simulation against the real-fitted model, with gradients passing through a Kalman-style filter.
The approach rests on two claims. First, for active tracking, much of the sim-to-real gap that matters for control lives in where detections happen, not in pixel appearance, and a model with a handful of parameters can capture it. Second, once detection probability depends on state, a missed detection carries information ("the target is probably not where I expected, because I would have seen it there"). A filter that simply skips the update on a miss throws that information away. The planner should exploit it, and only a smooth, state-dependent detection model makes that possible.
Core Claim
The detection model is the geometric soft cone multiplied by a learned degradation factor:
$$p_d(x, s) = \Phi\!\left(\frac{d_{\text{FoV}}(x,s)}{\tau}\right)\,\sigma\!\left(w^\top \phi(x,s)\right).$$
Here $x$ is the target state and $s$ is the sensor pose and velocity. $d_{\text{FoV}}$ is the signed distance from the target to the edge of the viewing cone, $\Phi$ is the Gaussian cumulative function with sharpness $\tau$, and $\sigma$ is the logistic function. $\phi(x,s)$ is a short feature vector containing range, viewing elevation, camera angular rate (a proxy for blur), signed distance to the nearest occluder along the line of sight, and a weather indicator. The weights $w$ are fit to logs of whether the actual detector fired.
The planner minimizes the expected entropy of the target belief after the next measurement. That expectation is
$$\mathbb{E}[H] = \bar p_d\, H(b^{+}_{\text{det}}) + (1-\bar p_d)\, H(b^{+}_{\text{miss}}),$$
where $\bar p_d$ is $p_d$ averaged over the current belief and $b^{+}$ are the posterior beliefs after a detection or a miss. The missed-detection posterior reweights the prior by $1-p_d$. It is not Gaussian, so we approximate it by moment-matching over sigma points (a fixed set of deterministic samples of the belief). That approximation is differentiable in $s$.
What is new is not smoothing the field of view. That is already done in Policy Learning for Active Target Tracking over Continuous SE(3) Trajectories and Learning Continuous Control Policies for Information-Theoretic Active Perception. What is new is three things together. The visibility function is fit to a real sensor and then swapped into simulation training. The planner differentiates through informative misses. And we test whether this transfers on a sim-to-real tracking benchmark.
Relation to Prior Work
Two papers supply the method we reuse. Policy Learning for Active Target Tracking over Continuous SE(3) Trajectories backpropagates target entropy through Kalman filter updates using a probit over the field-of-view signed distance. Its model-based gradients held an episodic reward near 5.41 in a 7-target scenario, while PPO fell to 1.94. It was tested only in a simulated planar world with Gaussian noise and a purely geometric cone. The earlier Learning Continuous Control Policies for Information-Theoretic Active Perception states that it assumes clear lines of sight and perfect association, and lists real deployment as future work. We keep their gradient machinery and replace the geometric cone with a calibrated, state-dependent detector model.
The closest competitor in spirit is Probabilistic Visibility-Aware Trajectory Planning for Target Tracking in Cluttered Environments. Its Belief-space Probability of Detection is the chance that a Gaussian target belief falls inside the unoccluded cone. When that probability is low, it inflates the filter's covariance, and it ran on real ground robots at 10 Hz. It differs from our proposal in three ways. Its notion of detection is still geometric: uncertainty enters through the belief, not through how the sensor degrades. It treats a missed detection as covariance inflation rather than as a measurement. And its sensor model is not fit to data from a real sensor. Its authors also list non-Gaussian extensions as open. Our missed-detection update is exactly such an extension.
The sim-to-real side constrains and motivates the design. Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation showed that clean synthetic scans give learners a shortcut. Real sensor dropout is structured and depends on state, and learned dropout masks help, raising SemKITTI from 15.2% to 22.8% mIoU. This is evidence that the structure of dropout matters, but it was shown only for offline segmentation. We carry the same insight into closed-loop control, with an explicit, interpretable dropout model in place of an adversarial mask.
Deep Drone Acrobatics (Extended Abstract) showed that abstracting raw pixels into features transfers zero-shot: 100% success, against 0% for raw-pixel policies. It also notes that the abstraction collapses exactly when features disappear. Our detection model is a different abstraction, and it is designed to describe when the abstraction fails.
The benchmark and the main baseline come from Open-World Drone Active Tracking with Goal-Centered Rewards and A Cross-Scene Benchmark for Open-World Drone Active Tracking. They train model-free reinforcement learning with ground-projected rewards and a curriculum across 24 Webots scenes, and they transfer to a real DJI Mini 3 Pro. Their policy keeps no explicit belief over the target and no visibility model. When the target vanishes, it has nothing to reason with.
Research Program
The first stage is calibration. We fix one detector-plus-tracker front end and run it on two sources. The first is simulated DAT rollouts across the Day, Night, Foggy and Snow conditions. The second is the real aerial videos on which GC-VAT reported transfer results (VOT, DTB70, UAVDT). On the real videos, range, elevation and occlusion are only partly observable. We estimate range from box scale using known target class sizes, estimate angular rate from global image motion, and take occlusion from dataset annotations where they exist. On each source we fit $w$ by logistic regression and record the held-out log-likelihood of detection events.
This stage alone answers a factual question no paper in our file addresses: how differently does the same detector's firing map look in simulation versus reality, and along which state dimensions does it differ?
The second stage trains the policy. We train the belief-space policy in DAT with gradients through the filter, as in Yang et al., extended to the drone's camera and yaw in 3D. During training, detections are sampled from the real-fitted $p_d$ rather than from the simulator's own detector. This is the swap that moves the gap from pixels to a low-dimensional model.
The third stage is deployment: in held-out DAT scenes and weather conditions, on the real video replay protocol, and in real flights with the same drone class as GC-VAT if hardware access permits.
Baselines
The naive reference is the same filter-gradient policy using the purely geometric probit cone. Its only robustness comes from additive Gaussian measurement noise and independent random dropout at a tuned rate. This is what the visibility literature does today. Every improvement we claim must be measured against it under identical training steps, network size and scenes.
The strongest existing method is GC-VAT (Open-World Drone Active Tracking with Goal-Centered Rewards). It reports a 72% Tracking Success Rate on DAT in simulation, against 20–30% for AOT and D-VAT. It reports a Correct Action Rate above 80% on real-video transfer and an 88.4% success rate in real flight without fine-tuning. We will run its released code on the same scene splits. We will give it the same detector front end where its architecture allows, and the same number of environment interactions. If that budget is below what it used originally, we will also report it at its own original budget. Beating it means higher Tracking Success Rate on held-out scenes. It especially means higher success on the Night, Foggy and occlusion-heavy subsets and on the real-video protocol, because that is where a calibrated miss model should help. Matching its overall average while using far fewer samples would count as a partial result, not a win.
The belief-space planner of Probabilistic Visibility-Aware Trajectory Planning for Target Tracking in Cluttered Environments is a second strong competitor. It reports 94% success under high noise, against 48% for a deterministic planner. We will reimplement it in the DAT setting with its covariance-inflation rule, give it the same occupancy information our planner sees, and tune its noise parameters on the same calibration logs.
Two ablations isolate the ingredients. The first keeps the real-fitted $p_d$ but uses a filter that skips the update on a miss. This isolates the value of negative information. The second keeps the full method but fits $p_d$ on simulated rather than real logs. This isolates the value of real calibration as opposed to merely having a richer model.
Why not just…?
Why not just add heavy random dropout and noise in simulation and train PPO? Independent dropout cannot express that misses cluster at long range, during fast yaw, and behind foliage. Adversarially Masking Synthetic to Mimic Real found that unstructured corruption leaves learners exploiting the wrong signal. Model-free policies also collapse as tracking becomes harder, falling from 5.41 to 1.94 episodic reward in Policy Learning for Active Target Tracking over Continuous SE(3) Trajectories. Our naive baseline is exactly this randomization, applied to both the gradient-based and the PPO learner, so the experiment measures the gap directly.
Why not just shrink and tune the geometric cone? This is the real threat. A deterministic cone with a fitted maximum range and a fitted edge sharpness might capture most of what matters, and our model would then add only parameters. We will fit this tuned cone on the same real logs with the same likelihood objective and compare held-out detection log-likelihood and closed-loop success. If the tuned cone matches us, we will report that state-dependent degradation does not matter for this task, and the contribution narrows to the missed-detection update.
Why not just use the probability-of-detection planner with inflated Gaussian noise? Inflating covariance tells the planner it is uncertain, but not where the target is likely to be after a miss. Its own authors flag non-Gaussian noise as unaddressed. The comparison against this planner in occlusion-heavy DAT scenes, where misses are frequent, settles whether the extra structure pays off.
Why not just fine-tune on real data? Closed-loop real rollouts for active tracking are expensive and risky. Our method uses only passive detector logs from existing videos. If a small amount of real fine-tuning of GC-VAT beats us, that would still leave our method as the cheaper option, and we would report both.
Proposed Experiments
The first decisive check is offline. Does the real-fitted $p_d$ predict real detector firing better than the geometric cone and the tuned cone, measured by held-out log-likelihood and Brier score? And do the simulated and real fits differ significantly along identifiable features, with blur and range expected to dominate? If the real and simulated fits are indistinguishable, the core premise fails and we stop.
The second check is closed loop in DAT. We train on Day scenes and evaluate on held-out scenes and on the Night, Foggy and Snow conditions. In the shifted conditions, $p_d$ is refit from a small number of passive logs and the policy is not retrained. We report Tracking Success Rate against GC-VAT's 72% and the baselines above. We also report time-to-reacquire after loss, because informative misses should shorten it.
The third check is the real-video Correct Action Rate protocol used by GC-VAT, against its reported figure above 80%. If hardware is available, real flights follow, against its 88.4% success.
Throughout, the negative-information ablation and the simulated-versus-real calibration ablation tell us which ingredient produced any gain. A result in which only the real-calibrated, miss-aware variant improves on the occlusion-heavy and adverse-weather subsets would support the core claim.
References
- Policy Learning for Active Target Tracking over Continuous SE(3) Trajectories
- Learning Continuous Control Policies for Information-Theoretic Active Perception
- Probabilistic Visibility-Aware Trajectory Planning for Target Tracking in Cluttered Environments
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Deep Drone Acrobatics (Extended Abstract)
- Open-World Drone Active Tracking with Goal-Centered Rewards
- A Cross-Scene Benchmark for Open-World Drone Active Tracking