Planning Flights Around What the Camera Will Actually See

A camera-guided drone that flies aggressively tends to blind itself. A quadrotor has to tilt to accelerate, so a fixed camera swings away from the landmarks its position estimate depends on. Fast rotation also smears the image, which destroys the feature matches the estimator needs. A line of work called perception-aware trajectory optimization (planning motion so the sensor keeps seeing useful things) has shown large gains. In a dark room lit only by four LEDs, APACE succeeded on every flight, while perception-agnostic planners failed every time and drifted more than 10 m. These planners share a quiet assumption, though. They model "seeing" as geometry: a landmark counts as useful if it lies inside the field of view and near the image center. Whether the real sensor pipeline would track that landmark through blur, underexposure, rolling-shutter warp and readout noise is not part of the model.

We propose to replace that geometric proxy with a learned trackability critic. This is a small network that predicts whether the onboard visual-inertial odometry (VIO: a position estimator that fuses tracked image features with inertial measurements) will keep enough features alive along a candidate trajectory segment. It is trained entirely in simulation under wide randomization of sensor physics. The critic runs in a slow planning tier that reshapes the trajectory and also passes a per-segment "blur budget" down to a fast model predictive controller. Domain randomization is normally used to make a policy tolerant of noisy inputs. Here it is used to teach the planner which motions produce noise in the first place.

Core Claim

For agile vision-based flight, the dominant sim-to-real failure is not that the controller mishandles corrupted inputs. The failure is that the planner commands motions that corrupt the inputs. A perception cost learned from sensor-randomized simulation will transfer zero-shot to hardware and will keep VIO alive under degraded imaging where geometric perception costs do not, without the planner having to fly timidly. The claim is falsifiable in one specific way. If a hand-tuned geometric cost with an angular-rate penalty matches the learned critic across a sweep of real imaging degradations, the claim is wrong.

Where This Sits

Three strands of prior work motivate the gap. Perception-aware optimizers treat perception as geometry. PAMPC puts the camera projection into a 100 Hz model predictive controller as soft costs on a landmark's image-center offset and image-plane velocity. APACE uses sigmoid-smoothed field-of-view checks, with separate position and yaw stages that optimize covisibility and parallax. RAPTOR adds active yaw for information gain. GFM-Planner scores LiDAR localizability from a Hessian but assumes ideal registration. All four assume that a visible landmark is a tracked landmark. We keep their optimization machinery and change what the perception term measures.

Two further papers supply the method. The second strand is Deep Drone Acrobatics. It distills a privileged simulated controller into a student that sees VIO feature tracks rather than pixels, and reports 100% real-world success against 0% for pixel-trained policies. That result also constrains our design: learned components should consume geometric abstractions, not raw appearance. Its expert, however, is perception-agnostic, and the paper leaves open that the policy fails when the feature front-end collapses. Our proposal targets exactly that collapse. Third, Asymmetric Actor Critic shows that a critic with privileged simulator state can train a deployable module under randomization. We borrow that pattern: the critic's training labels come from privileged simulation, while its inputs are only quantities available at plan time.

The hierarchical split follows HiRT in spirit: a slow module produces guidance and a fast module stays stable. The two tiers here are a roughly 10 Hz learned planner and a 100 Hz optimizer rather than a vision-language model and a motor policy. This also addresses the hard-versus-soft tension. PCMPC argues for hard visibility constraints, while PAMPC argues that hard constraints cause solver infeasibility. Our slow tier sets constraint levels based on predicted trackability, so the fast tier can stay feasible.

Research Program

We would build a sensor-degradation layer on top of Flightmare, whose decoupled physics lets us run many vehicles in parallel. The layer renders motion blur from the true angular and linear velocity during a randomized exposure time, rolling-shutter warp, exposure and gain changes, readout noise, and lens vignetting. The real VIO front-end then runs on those corrupted frames. Each simulated flight segment is labeled with privileged outcomes: the number of features that survive, and the VIO position error growth over the segment.

The trackability critic $T_\phi(s, \sigma)$ outputs the predicted probability that at least $k$ features survive segment $s$. Its inputs are a geometric summary of the segment ($s$ contains the planned body rates, accelerations and the projected landmark set from the current map) and $\sigma$, the current camera settings (exposure, gain). It sees no raw pixels. This follows the abstraction lesson from Deep Drone Acrobatics and should aid transfer.

The slow tier optimizes a B-spline trajectory $\tau$ made of segments $s_1,\dots,s_N$ by solving

$$\min_{\tau} \; t_f(\tau) + \lambda \sum_{i=1}^{N} \big(1 - T_\phi(s_i, \sigma)\big) + J_{\text{coll}}(\tau),$$

where $t_f$ is flight time, $\lambda$ trades speed against predicted trackability, and $J_{\text{coll}}$ is the usual collision cost. We use gradients of the differentiable critic, starting from an APACE-style warm start.

The optimized segments are then passed to a PAMPC-style controller together with a per-segment bound on body rate. We call this bound the blur budget. It is the largest rate at which $T_\phi$ stays above threshold. The fast tier enforces it as a softened constraint with a slack variable. When predicted trackability is high the budget is loose and the vehicle flies fast. It tightens only where the critic expects the camera to go blind.

Finally, we would study the randomization width directly. Wide sensor randomization could make the critic pessimistic and the drone timid, a known failure mode (How to pick the domain randomization parameters). We therefore compare wide and narrow ranges and check how well the critic's predictions match real outcomes.

Baselines

The naive reference point is a perception-agnostic minimum-time spline tracked by a standard controller with independent yaw pointing. In APACE's dark-room arena this configuration failed 100% of the time with more than 10 m of drift (APACE). It fixes the floor of the comparison.

The strongest existing method is APACE itself, the geometric perception-aware planner, tracked by the same fast controller. It is the strongest competitor because it already optimizes feature matchability and parallax. Its reported numbers are 100% success with 0.12–0.14 m final goal error on the physical four-LED arena, and a position RMSE reduction "up to an order of magnitude" in AirSim (APACE). At nominal imaging we expect to tie it, since it already reaches 100% there. Beating it therefore means one of two outcomes. The first is higher success or lower VIO error at matched flight time under degraded imaging. The second is equal success at shorter flight time. For degraded imaging, no reported result was found on Lacuna for APACE or PAMPC.

A second strong baseline augments APACE with PAMPC's image-plane velocity penalty (PAMPC) plus an explicit body-rate penalty. We tune it with the same number of simulated evaluation flights that the critic consumes, using a grid over weights, so both arms get equal access to the degraded simulator.

A third baseline is a Deep-Drone-Acrobatics-style student. It is distilled from our planner-plus-controller teacher under the same sensor randomization and consumes feature tracks and inertial data. It tests whether a single end-to-end policy can absorb the benefit.

Ablations isolate the key ingredients. The first trains the critic on clean simulation with no sensor randomization. The second keeps the critic in the slow tier but removes the blur budget passed to the fast tier. The third varies randomization width from narrow to wide.

Why not just…?

Why not just add a hand-tuned angular-rate or image-velocity penalty to PAMPC or APACE? This is the real threat. Blur is largely a function of angular rate, and PAMPC already penalizes image-plane velocity. What a fixed penalty cannot represent is interaction. The same rate is harmless in a bright, textured scene with short exposure and fatal in a dim one with long exposure. Which landmarks survive also depends on their depth and image position, not only on body rate. So we tune this baseline with the full matched budget and evaluate across an exposure sweep. If one tuned weight setting matches the critic across all degradation levels, the learned critic is unjustified and we would report that.

Why not just distill everything into one end-to-end policy with sensor randomization, as Deep Drone Acrobatics does? Doing so makes the controller tolerant of degraded tracks. When tracks vanish entirely, that policy's inputs vanish too, which is the failure the file attributes to it in dark or featureless scenes. The distilled-student baseline measures directly whether a policy can learn on its own to avoid blinding motions. We expect it to do worse in dark scenes, but a student distilled from our own teacher could plausibly close the gap. If it does, the result would still show that a trackability-aware teacher is the necessary ingredient.

Why not just make the front-end robust instead, for example with a randomized-trained feature tracker or inertial fallback? A more robust front-end raises the level of degradation at which tracking collapses, but it does not stop the planner from commanding motions past that level. We treat front-end robustness as orthogonal and run every arm, including the baselines, with the same best available front-end. We then test whether the planning gain survives on top of it.

Proposed Experiments

In simulation, we use APACE's textureless AirSim room and Flightmare's Warehouse and Garage scenes. We sweep exposure time, illumination and noise level at test values drawn outside the training randomization range. We report success rate, VIO position RMSE, surviving feature count, and flight time. The central plot is success against degradation severity for each arm. The prediction is that geometric planners hold at nominal settings and fall off steeply as blur grows, while the critic-based planner degrades gracefully and pays at most a small time penalty.

On hardware, we replicate the APACE four-LED dark-room protocol and the PAMPC sparse-light room. Degradation is set with physical knobs: camera exposure time, LED dimming and flight speed. Every arm uses the same airframe, VIO and controller.

Two decisive checks follow. The first is calibration: on real flights, compare the critic's predicted feature survival with measured survival. Good calibration on hardware from a critic that never saw real data would support the zero-shot claim. A clean-sim critic that stays well calibrated would instead show that sensor randomization is unnecessary. The second is the matched-budget comparison against the tuned rate-penalty baseline under the exposure sweep. It settles whether the learned cost adds anything beyond one well-chosen scalar. Finally, the width ablation shows whether wide randomization makes the drone timid, measured as flight time at equal success. That tells us where the blur budget helps and where it overconstrains.