End-to-end control policies map raw sensor streams straight to motor or velocity commands with a single neural network. They are usually trained in simulation with domain randomization, meaning the simulator's textures, lighting, sensor noise and physics are scrambled across episodes so the real world looks like just one more variation. These policies typically see two kinds of input: camera or depth images, and the robot's own motion state (velocity, attitude, inertial readings), which we call ego status. We argue that the two ingredients work against each other. Visual randomization deliberately makes the image stream harder to read, while ego status stays clean and correlates strongly with the next action. The network is therefore pushed to steer mostly from its own velocity and to treat vision as optional. The result is a policy that scores well in simulation but is quietly half-blind when it matters.
We propose to measure this failure in fast vision-based quadrotor obstacle avoidance and to fix it with scene-swapped relabeling. In simulation we take the drone's ego history from one trajectory, place it in a different obstacle scene, render what it would see there, and ask the privileged expert what it would do. Training on these recombined samples removes any correlation between ego status and the obstacle-driven part of the action. The policy keeps full access to its own dynamics, but it can only predict the expert by looking.
Core Claim
In behavior-cloned visuo-proprioceptive flight policies, strong visual domain randomization lowers how much the policy actually uses vision, and this shows up as higher collision rates at high speed and in cluttered scenes. Scene-swapped relabeling restores vision use without removing or corrupting ego status. It should beat the obvious fix, randomly dropping or noising ego status, because that fix also destroys the history information the policy needs to infer unknown dynamics.
Grounding in Prior Work
The shortcut was demonstrated in open-loop driving. Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? shows that on nuScenes a model with no visual input at all reaches 0.35 m average L2 error, against 0.46 m for the full vision model UniAD. The same paper finds that blanking the cameras barely hurts planning, while perturbing ego velocity is catastrophic. That paper motivates our work, but it studies open-loop driving logs without domain randomization. We study closed-loop control trained under randomization, where the incentive to ignore vision should be stronger.
Three lines of work supply our setting and method. Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidance trains a ViT+LSTM policy by behavior cloning from a privileged expert in Flightmare. Its inputs are depth images, attitude and forward velocity, which is exactly the mixed input that invites the shortcut. It flies zero-shot at up to 7 m/s, but its training is synchronized and delay-free, and it does not analyze how much the policy relies on vision. Learning Agile, Vision-based Drone Flight: from Simulation to Reality supplies the privileged-expert-to-student recipe with randomization that our relabeling depends on: we reuse its expert and change which states it labels. Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization shows that randomizing the separate delays of the vision and proprioception streams nearly doubled real-world distance traveled (859.9 cm vs 444.7 cm). We include that delay randomization as standard practice, because a vision-reliance fix that breaks under realistic latency would be meaningless.
Two results constrain the design. Understanding Domain Randomization for Sim-to-real Transfer proves that under randomization the policy must have memory so it can infer the hidden physics. This rules out the tempting fix of simply deleting ego status or its history, and it is why we keep the LSTM and full ego input. Robust Policies via Mid-Level Visual Representations reports that visual randomization can cause training collapse and that scratch-trained pixel policies reached 0% on RLBench Pick & Place, compared with 100% for frozen surface-normal features. We read that collapse as possibly the same phenomenon we target, seen from the optimization side, and we test the frozen-feature route directly as a competitor. (CAD)²RL (Real Single-Image Flight without a Single Real Image) avoids the problem by feeding no ego status at all. It pays for that by outputting only high-level velocity commands through a low-rate autopilot and by suffering monocular scale ambiguity.
Research Program
The work has three parts.
The first part is a diagnostic. We define a closed-loop vision-reliance score from two collision rates. The first is the policy's collision rate when its depth frames are replaced online by frames from an unrelated episode rendered at matched ego state. The second is its collision rate with its true frames. The score is the first minus the second. A policy that truly steers by vision should crash far more with swapped frames; a shortcut policy should barely notice. We also report the reverse probe from the driving study: collision rate when the velocity estimate receives a small bias. We compute both probes for policies trained across a sweep of visual randomization strength. The sweep varies depth noise, stereo-style holes, texture and lighting (for an RGB variant), and delay randomization. This tests the first half of the claim: vision reliance should fall as randomization strength rises.
The second part is the method. During data aggregation, for each student rollout state we sample a donor scene. We keep the drone's ego history, including attitude, velocity and delayed IMU buffers. We re-place the vehicle at the same relative pose inside the donor scene, render the randomized depth sequence it would have seen, and query the privileged expert for its action there. These relabeled samples are mixed into training at a ratio $\rho$ (the fraction of each batch that is scene-swapped), which we sweep. Two safeguards keep the recombined states sensible. We reject swaps that start inside an obstacle. We reject swaps whose expert action requires dynamics the ego history makes infeasible, for example a sharp reversal at 10 m/s, so that no physically impossible targets are taught. The same recipe carries over to an RGB input variant.
The third part keeps the dynamics question in scope. We add randomized unmodeled aerodynamic moments of the kind End-to-end neural network based optimal quadcopter control identified as the dominant reality gap for its state-based controller. This checks that relabeling does not make the policy ignore the ego history it needs for dynamics inference. The main output stays as velocity commands, matching the ViT paper. A secondary body-rate output is tried only if the main result holds.
Baselines
The naive reference is the ViT+LSTM student trained by behavior cloning with no domain randomization. It anchors how much any randomization helps.
The strongest existing method is the ViT+LSTM policy of Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidance, retrained with our full visual and delay randomization and the same expert, dataset size, architecture, optimizer budget and hyperparameter search. The file contains no numeric collision rate for it. It reports only that it beat the expert and the ConvNet and UNet baselines above 5 m/s in "Spheres", and zero-shot real flight up to 7 m/s, so we will reproduce it ourselves rather than quote a number. Beating it means a lower collision rate at 7 and 10 m/s in both "Spheres" and the held-out "Trees" environment. The reduction must be statistically significant across at least five seeds, and it must come with a clearly larger vision-reliance score.
Ego-status dropout and noise is the most important competitor. During training, ego inputs are zeroed or heavily noised with a tuned probability, given the same tuning budget as our $\rho$. It is the cheapest way to force vision use.
A blind ego-only LSTM trained on the same data plays the role of the driving study's Ego-MLP. It shows how much of the task can be solved without looking at all.
A frozen mid-level depth or normals encoder, in the spirit of Robust Policies via Mid-Level Visual Representations, is run for the RGB variant.
Standard DAgger without scene swapping isolates our key ingredient. It relabels off-distribution states but keeps each scene paired with its own ego history. Our method differs from it only in the swap.
Why not just…?
Why not just drop out or noise the ego-status input? This is a real threat. If it matches our method on collision rate and on the velocity-bias probe, the method adds nothing and the contribution reduces to the diagnostic. We expect it to fall short under aerodynamic-moment randomization. The theory in Understanding Domain Randomization for Sim-to-real Transfer says the policy needs its history to infer hidden physics, and dropout withholds exactly that history. The dropout-versus-swap comparison, run with and without moment randomization, settles the question.
Why not just remove ego status, as (CAD)²RL does? A memoryless or ego-free policy cannot infer latent dynamics, and (CAD)²RL could only fly slow hallway missions through an autopilot. Our ego-free ablation within the sweep will show how that tradeoff plays out at speed.
Why not just use a frozen mid-level encoder so vision is easy to use? That route addresses the training collapse, but it does not remove the incentive to prefer clean ego status, and it has not been shown for delay-randomized agile flight. It is included as a baseline. If it achieves a vision-reliance score as high as ours, that is evidence the problem is feature difficulty rather than label correlation.
Why not just use DAgger? DAgger fixes compounding errors but keeps each scene tied to its own ego trajectory, so the shortcut correlation survives. The DAgger baseline measures precisely that.
Proposed Experiments
All training and simulated evaluation run in Flightmare on the DodgeDrone "Spheres" (training) and "Trees" (held-out) environments. Speeds are 3, 5, 7 and 10 m/s, with randomization strength swept from none to extreme. Metrics are collision rate, the vision-reliance score, collision rate under velocity-estimate bias, and success under injected aerodynamic moments.
Three decisive checks follow. First, the baseline's vision-reliance score must fall monotonically with randomization strength, and our method's must not. If the baseline's score does not fall, the premise is wrong and we report that. Second, our method must beat both the reproduced ViT+LSTM and ego dropout at 7–10 m/s in "Trees". Third, it must not lose to ego dropout's robustness under moment randomization.
If hardware is available, zero-shot flights on a Falcon250-class quadrotor at up to 7 m/s would compare our policy with the reproduced baseline, including a real frame-freeze probe.
References
- Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
- Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidance
- Learning Agile, Vision-Based Drone Flight: From Simulation to Reality
- Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
- Understanding Domain Randomization for Sim-to-real Transfer
- Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
- CAD2RL: Real Single-Image Flight Without a Single Real Image
- End-to-end neural network based optimal quadcopter control
- Flightmare: A Flexible Quadrotor Simulator