Learned quadrotor controllers that fly aggressive trajectories in the lab almost always cheat in one specific way. They read position and velocity from an external motion-capture system: a ring of cameras that tracks reflective markers on the drone at hundreds of hertz with millimetre accuracy. Take the cameras away and the drone must estimate its own state from onboard sensors. The cheapest and most reliable of these is the inertial measurement unit (IMU), which combines an accelerometer and a gyroscope. An IMU works in smoke and darkness, but it drifts. A small gyroscope bias tilts the estimated gravity direction, and integrating that tilt twice makes position error grow with the cube of time. We propose to stop asking the low-level controller to track goals in a global frame that drifts underneath it. Instead, a slow planner issues short relative subgoals, such as "move this displacement in a gravity-aligned frame anchored where you are now." A small recurrent policy flies directly from raw IMU and motor-speed signals. The planner holds each subgoal only as long as the inertial drift budget allows, and then re-anchors.
The point is that cubic drift is only catastrophic over long horizons. If the reference frame is reset every $P$ seconds, the error the controller sees is set by $P$ and not by the total time since the last camera fix. In this design, the decision of how long to hold a subgoal is computed from inertial physics instead of tuned by hand. Within the hierarchy, the slow tier owns global position, and the fast tier owns only what an IMU can actually observe over a short window.
Core Claim
A hierarchical flight stack can run agile trajectories with no external localization, and degrade gracefully under multi-second visual dropouts, if it meets three conditions. First, the interface between tiers carries only quantities that are observable from inertial data over short windows. These are roll and pitch (anchored by gravity), velocity in a gravity-aligned frame, and short relative displacements. Second, the low-level executor is a recurrent policy trained in simulation against randomized IMU bias, so it learns to act well despite its own estimation error rather than consuming an estimate passively. Third, the subgoal persistence period is set by a drift budget:
$$P^\star = \max \{ P : \tfrac{1}{2}\sigma_a P^2 + \tfrac{1}{6} g\,\sigma_\omega P^3 \le \varepsilon \}.$$
Here $\sigma_a$ and $\sigma_\omega$ are the current accelerometer and gyroscope bias uncertainties, $g$ is gravity, and $\varepsilon$ is the positional tolerance the task can absorb. The two terms are the standard double-integrated accelerometer bias and triple-integrated gravity-leak terms. In practice we replace the constants with a learned covariance, as TLIO does. $P^\star$ is also floored, because too-frequent re-planning is itself harmful (see below).
The testable prediction is specific. When vision is lost, the executor's error relative to its current subgoal should scale with $P^3$ and stay flat in the time elapsed since the last camera fix. A global-frame modular stack should show error growing with elapsed time.
Where This Sits
Four bodies of work motivate the proposal.
SimpleFlight (What Matters in Learning A Zero-Shot Sim-to-Real RL Policy for Quadrotor Control?) trains a feedforward reinforcement-learning actor that outputs collective thrust and body rates and transfers zero-shot to real quadrotors. Its hardware trials depend on motion capture, and its authors name onboard sensing as the main open problem. We reuse its training recipe: rotation-matrix inputs, action-smoothness penalties, and selective domain randomization. We replace motion-capture state with raw inertial input.
DIDO (DIDO: Deep Inertial Quadrotor Dynamical Odometry) shows that IMU plus rotor tachometers suffice for low-drift quadrotor dead reckoning. It uses a de-biasing network, a recurrent velocity-position network, and a two-stage filter that separates attitude from translation. Its ablation shows yaw drift ruins position without de-biasing. DIDO supplies the method for our estimator baseline and our attitude/translation split. It is evaluated only as a passive estimator on smooth trajectories, whereas we close the control loop.
TLIO (TLIO: Tight Learned Inertial Odometry) and the legged-robot work of Buchanan et al. regress short-window displacement with a calibrated covariance. TLIO reports over 99% of errors inside predicted 3σ bounds. That calibration is what lets us compute $P^\star$ online.
Qin et al. optimize only position and yaw in their pose graph because gravity makes roll and pitch observable. That observation dictates which quantities may cross our interface.
On the hierarchy side, HiRT (HiRT) runs a slow vision-language model asynchronously above a 10 Hz policy. It trains the fast tier on stale latents and raises dynamic-task success from 48% to 75%. HiRT works on slow tabletop arms with purely visual state and never faces a drifting frame.
When to Re-Plan constrains our design. On abstract reasoning tasks it finds that re-planning every step is worse than no hierarchy (loss 1.674 vs. 1.640), with an optimum at a persistence of 3 (1.544). Inertial drift pushes toward short persistence and that result pushes against it, which is why $P^\star$ has a floor. That paper fixes $P$ by sweep. We make $P$ a function of measured uncertainty, and we test the trade-off in a physical regime where its result has never been checked.
Selective Sensor Fusion shows that a recurrent network can learn to drop corrupted visual input and lean on inertial features. It remains an offline estimator. We borrow its corruption protocol for training and evaluation.
Finally, the survey by Li et al. warns that black-box recurrent inertial models can learn spurious drift cancellation that breaks out of distribution. This is the strongest argument against our recurrent executor. The frame reset is our answer to it: the policy never has to remember drift beyond one window, so there is less room for spurious long-horizon shortcuts.
Research Program
We build a two-tier stack.
The slow tier runs at 1–5 Hz. It holds the global trajectory and fuses sparse visual fixes when they exist, using standard visual-inertial odometry. When vision is lost it falls back to propagation with a DIDO-style estimator. At each re-plan it expresses the next segment of the reference trajectory as a relative displacement and velocity profile in a gravity-aligned frame anchored at the current pose. It then computes $P^\star$ from the estimator's covariance and holds that subgoal for $P^\star$. If $P^\star$ shrinks below the floor during a long outage, the slow tier switches to a conservative mode, either hover or retrace, instead of issuing subgoals it cannot track. Communicating degraded certainty back to the planner in this way is something no existing hierarchy in the file does.
The fast tier is a small gated recurrent unit (GRU) policy running at 200–500 Hz. It receives raw IMU readings, motor speeds, the current subgoal, and time since the frame reset, and outputs thrust and body rates. Its hidden state is reset or re-conditioned at each subgoal boundary.
We train the fast tier with SimpleFlight's recipe in simulation, randomizing IMU bias as a random walk with occasional jumps that mimic throttle and thermal effects, along with vibration noise and motor lag. Reward is tracking of the relative subgoal. Throughout, motion capture serves only as ground truth for scoring, never as an input.
The analysis has two parts. We measure how within-window error scales with $P$ and with elapsed outage time. We also stress-test the recurrent executor with aggressive maneuvers outside its training distribution, which addresses the concern raised by Li et al.
Baselines
The naive reference point is SimpleFlight's feedforward actor, retrained identically but fed state from onboard visual-inertial odometry (OpenVINS) in place of motion capture, tracking global-frame goals. This is what a practitioner would do first. On the contact benchmark, Zhan et al. report OpenVINS velocity error of 0.0619 m/s, against 0.0121 m/s for their contact-aware estimator, which indicates the noise level this baseline inherits even with vision. This naive baseline must collapse under dropouts for our motivation to hold.
The strongest competitor is a modular stack. It pairs a DIDO-style estimator (de-biasing network, recurrent velocity-position network, two-stage filter), trained on the same flight data, with the same SimpleFlight actor tracking global goals at a fixed re-planning rate swept over the same values our $P^\star$ can take. DIDO is the strongest inertial-only quadrotor estimator in the file and directly targets our sensors. The file reports DIDO beating TLIO, RoNIN and a dynamics UKF on absolute trajectory error, but no reported numeric ATE was found on Lacuna for DIDO, so we will reproduce it on its own dataset first.
Both systems get the same simulator, the same randomization, the same number of training environment steps, and the same hyperparameter search budget. Beating this baseline means a lower crash rate and lower ground-truth tracking error during 2–10 s visual outages at speeds of 3 m/s and above, with parity when vision is available.
Two ablations isolate the ingredients. The first keeps our relative-subgoal interface and drift-budgeted $P^\star$ but uses the modular estimator-plus-MLP executor; it tests whether the recurrent executor matters. The second keeps the recurrent executor but feeds it global-frame goals at fixed persistence; it tests whether the interface matters. A third compares drift-budgeted $P^\star$ against a hand-tuned fixed $P$ and against a simple covariance-threshold trigger.
Why not just…?
Why not just use a classical visual-inertial estimator and a good controller? When vision is healthy this may be enough, and our parity condition says so. The file shows these estimators fail precisely in our target regime. MSCKF failed completely on corrupted KITTI where selective fusion held 0.145 m error (Selective Sensor Fusion). The filter-based ROVIO was more prone to divergence on long TUM VI sequences (TUM VI). Blackbird reports drift rising from 0.55% to 1.94% as speed goes from 1.12 to 3.08 m/s (Blackbird). The naive baseline measures this directly.
Why not just plug DIDO into SimpleFlight? This is the real threat, and it is the strongest baseline above. If the first ablation, the modular executor behind our interface, matches the full system, we will conclude that the contribution is the drift-budgeted relative interface and not the recurrent policy, and we will report it that way. What the plain DIDO-plus-SimpleFlight combination lacks is a mechanism that keeps global-frame drift from feeding into the controller's goals during long outages. The outage-length scaling test will show whether that matters.
Why not just re-plan as fast as possible, or trigger re-planning when covariance crosses a threshold? Fast re-planning shrinks drift per window, but When to Re-Plan found every-step re-planning worse than no hierarchy at all. Whether that holds for flight is unknown, and our persistence sweep tests it. A threshold trigger is a simpler cousin of $P^\star$; if it performs equally well we will adopt it, since the core claim concerns the relative interface and the drift-aware timing, not the exact formula.
Why not just train one end-to-end recurrent policy from IMU to motors with global goals? That is the second ablation. The survey by Li et al. predicts that such a policy will learn spurious drift cancellation and break outside its training distribution. The out-of-distribution maneuver test checks whether it does.
Proposed Experiments
Offline, we use Blackbird's aggressive flights. These provide a 100 Hz IMU, about 190 Hz motor speeds, 120 Hz cameras, and 360 Hz OptiTrack truth, plus FlightGoggles renderings that allow controlled visual dropouts. We also use the DIDO dataset. On both, we replay estimator and executor inputs to measure within-window displacement error as a function of $P$ and outage length, and to check how well the learned covariance predicts actual error, which is what makes $P^\star$ trustworthy.
In closed-loop simulation, all systems fly Blackbird-style and SimpleFlight-style figure-eights, zigzags and pentagrams at 1–7 m/s. Outages are injected at 0, 2, 5 and 10 s. The metrics are ground-truth tracking error, crash rate, and final error at outage end.
On hardware, a small quadrotor flies the same trajectories in a motion-capture arena that is used only for scoring, with cameras masked for outages. A second phase on a 900 m-scale outdoor loop uses TUM VI-style start-to-end truth. For context, BASALT reports 1.20 m ATE on the 918 m TUM VI loop with vision available.
The decisive checks are three. During outages, our executor's subgoal-relative error should stay flat in elapsed time and scale near $P^3$, while the modular baseline's error grows with elapsed time. The first ablation tells us whether the recurrent executor earns its place. Recovery after the planner enters conservative mode must succeed in the large majority of 10 s outages, measured as a crash rate below that of the strongest baseline at matched speed. If the first and third checks fail, the idea is wrong.
References
- What Matters in Learning a Zero-Shot Sim-to-Real RL Policy for Quadrotor Control? A Comprehensive Study
- DIDO: Deep Inertial Quadrotor Dynamical Odometry
- TLIO: Tight Learned Inertial Odometry
- Learning Inertial Odometry for Dynamic Legged Robot State Estimation
- Relocalization, Global Optimization and Map Merging for Monocular Visual-Inertial SLAM
- HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
- When to Re-Plan: Subgoal Persistence in Hierarchical Latent Reasoning
- Selective Sensor Fusion for Neural Visual-Inertial Odometry
- Inertial Sensing Meets Artificial Intelligence: Opportunity or Challenge?
- Aerial Manipulation with Contact-Aware Onboard Perception and Hybrid Control
- The TUM VI Benchmark for Evaluating Visual-Inertial Odometry
- The Blackbird Dataset: A Large-Scale Dataset for UAV Perception in Aggressive Flight