Bad Demonstrations, Good Observations: Map-Dropout Pretraining for Label-Efficient Aerial Navigation

Drones that follow language instructions through a city learn mostly by imitating human pilots, and human flights are expensive to collect. The best current systems also read a top-down map of the area, built from OpenStreetMap, alongside the drone's own camera. That map helps a great deal, but we suspect it also lets the policy cut corners. The policy can steer by the map and largely ignore what its camera sees. If so, it fails when the map is missing, stale, or misaligned. It also wastes scarce demonstrations, because the camera encoder receives little learning signal.

We propose two changes that work together. First, the map is randomly hidden during training. Second, while the map is hidden, the camera stream must predict the map's local content. This prediction task needs no human labels. It can be trained on cheap automatic flights, the same shortest-path trajectories that are known to be poor sources of actions. The core bet is simple: shortest-path flights are bad demonstrations but good observations. Self-supervision on them, gated by map dropout, should let a policy reach strong success with a small fraction of human flights and keep working when the map disappears.

Core Claim

On the CityNav aerial vision-and-language navigation benchmark, two ingredients should do better than either alone, and better than global cross-view feature alignment. The first is modality dropout: hiding the entire overhead-map input at random during training, rather than zeroing scattered features. The second is a cross-view prediction objective: from egocentric drone frames plus odometry, reconstruct the landmark and obstacle channels of the hidden map in the drone's local window. We expect the combination to win along two axes. With 5–10% of the human demonstrations, it should approach the success rate of the fully supervised map-using policy. It should also degrade gracefully, rather than collapse, when the map is removed or corrupted at test time.

What would be new is the split of supervision roles. Human flights supply actions. Abundant action-free flights supply the cross-view signal. Dropout is what forces the camera branch to actually absorb that signal.

Grounding in Prior Work

The motivating evidence comes from CityNav. It builds the CityFlight simulator over SensatUrban scans of Cambridge and Birmingham, collects 32,637 human flights, and introduces a Geographic Semantic Map (GSM): a top-down raster of landmarks, obstacles, explored area and candidate goals, fed as an extra input. Adding the GSM lifts AerialVLN success on unseen test areas from 56.6% to 72.7%, so the map carries a large share of the decision. CityNav also reports that the same architecture reaches 85.1% when trained on human demonstrations but 56.6% on shortest-path trajectories. That result is usually read as "synthetic data doesn't work." We read it more narrowly: shortest paths are poor action supervision, which says nothing about their value as observation data. CityNav always supplies the map and always uses the full demonstration set; both are exactly what we vary.

The closest prior method is Cross-View Policy Learning for Street Navigation. Working in a discrete street-grid RL setting, it randomly hides either the ground panorama or the aerial image ("view dropout"). It aligns the two views' global embeddings with an L2 loss and distills policies across views, raising zero-shot ground-view transfer success in Manhattan from 10.5% to 34.5%. We reuse its view-dropout idea but differ in three ways:

  • Our setting is continuous aerial instruction following learned by imitation, not grid RL.
  • Our cross-view signal is spatially structured prediction of map content, not alignment of global vectors.
  • We study the number of labeled demonstrations directly, which that paper does not.

Because its L2 alignment is a credible simpler substitute for our prediction objective, we run it as a baseline rather than dismissing it.

The modality-dropout literature supplies the mechanism and one constraint. M3 shows that masking whole modalities, rather than scattered features, is what breaks reliance on a dominant stream. HeMIS and mmFormer show that training on random subsets of inputs yields graceful degradation instead of failure on unseen zero inputs. MCE warns that the dropout rate interacts with how often modalities are missing in deployment, so we sweep the rate rather than fixing it. Compositional Failure in Audio-Visual LLMs finds that multimodal models collapse toward one modality's prior when the streams conflict. That motivates our stale-map test, in which the map disagrees with what the camera sees.

The label-efficiency literature supplies the expectation that domain-matched self-supervision cuts labeled data needs. For example, Autoregressive Sequence Modeling for 3D Medical Image Representation matches full supervision with 10% of the labels. On the remote-sensing side, How to Efficiently Adapt Foundation Models to Remote Sensing for Object Detection? finds that lightweight adapters on web-pretrained models transfer poorly to top-down imagery. This constrains us: we cannot assume an off-the-shelf encoder already relates oblique drone views to nadir maps, so that relation must be learned.

Research Program

The policy is AerialVLN with the GSM branch, as in CityNav, so that every gain is attributable to training rather than architecture. We add a small decoder on the egocentric branch. It takes the camera features accumulated over recent steps, together with the drone's known pose, and outputs the GSM's landmark and obstacle channels in a local window around the drone. We exclude the field-of-view and explored-area channels, which can be computed trivially from pose.

Let $D_L$ be the small set of human demonstrations and $D_U$ a large pool of action-free flights rendered in CityFlight along shortest paths and random waypoint tours. Let $m \in \{0,1\}$ be a map-visibility mask drawn with probability $p$ of hiding the map. Training minimizes

$$ \mathcal{L} = \mathbb{E}_{D_L}\big[\ell_{\text{act}}(\pi(o, m\cdot g, x), a)\big] + \lambda\, \mathbb{E}_{D_L \cup D_U}\big[(1-m)\,\ell_{\text{map}}(f(o, x), g_{\text{local}})\big], $$

where the symbols are as follows:

  • $o$ is the egocentric frame history and $x$ the language instruction.
  • $g$ is the GSM, $a$ the demonstrated action, and $\ell_{\text{act}}$ the imitation loss.
  • $f$ is the map-prediction decoder and $g_{\text{local}}$ the true local map patch.
  • $\ell_{\text{map}}$ is a per-cell cross-entropy, and $\lambda$ weights the two terms.

The factor $(1-m)$ means the prediction loss applies only when the map is hidden. This ties the auxiliary signal to exactly the condition in which the camera branch must stand alone. The unlabeled pool $D_U$ contributes only to the second term, because its actions are not trusted.

To touch the remote-sensing gap directly, we repeat the main experiments with a second overhead input. Instead of the symbolic GSM, we use a nadir RGB orthophoto rendered from the colored SensatUrban point cloud, with the same prediction targets. The question is whether egocentric-to-nadir prediction still helps when the overhead input is raw imagery rather than clean symbols. Extending the same recipe to camera dropout on the AIR-VLA aerial manipulation benchmark is a natural follow-up, but it is outside this note's scope.

Baselines

Naive reference. CityNav's AerialVLN+GSM, trained with no dropout and no auxiliary loss, on each demonstration budget (1%, 5%, 10%, 25%, 100% of the 32,637 flights). The full-data version is the strongest reported system. CityNav reports 72.7% success on test-unseen for AerialVLN+GSM. Separately, it reports 85.1% for AerialVLN+GSM trained on human demonstrations, versus 56.6% when trained on shortest paths. We reproduce these under our codebase before comparing and report our reproduced numbers next to the published ones.

Map dropout alone. The same policy trained with whole-map dropout at rate $p$ and no prediction loss. This is the cheapest version of the idea and the alternative most likely to make our prediction objective redundant.

StreetAir-style alignment. The recipe of Cross-View Policy Learning for Street Navigation, adapted to imitation: view dropout plus an L2 loss pulling the egocentric embedding toward the map embedding, trained on the same $D_L \cup D_U$. It is the strongest existing method for injecting cross-view signal, and it uses the same unlabeled pool.

Shortest-path action augmentation. The policy trained on human demonstrations plus shortest-path trajectories with their actions. This tests whether using the cheap flights as action supervision is as good as using them as observation-only supervision.

Ablations. - Prediction loss without dropout (the map is always visible; the decoder still predicts it). - Prediction loss on $D_L$ only, versus on $D_L \cup D_U$. This isolates the value of the cheap observation pool. - Prediction loss on observations from shortest paths, versus on observations from human flights with actions discarded. This tests whether observation coverage, rather than flight quality, is what matters.

Fairness and the bar to clear. All methods share the backbone, the number of training steps, and a fixed hyperparameter search of equal size; we sweep $p$ and $\lambda$ for our method and the corresponding weights for the baselines. Results are averaged over three seeds. Beating the field means two things:

  • At the 5% and 10% budgets with the map present, our method should exceed every baseline in success rate by more than the seed spread.
  • With the map absent, it should retain a clearly larger fraction of its map-present success than map-dropout-alone and StreetAir-style alignment.

Why not just…?

Why not just apply map dropout without the prediction loss? This is the real threat. M3 and HeMIS suggest dropout by itself already forces the camera branch to carry information. If dropout alone matches our method at every demonstration budget, the prediction objective is unnecessary and we will report that. Our hypothesis is that dropout alone helps robustness but not label efficiency. Without extra signal, the camera branch can learn only from the few action labels. The prediction loss is the only route by which the action-free pool $D_U$ enters training, so the gap should widen as the demonstration budget shrinks. The dropout-alone baseline and the $D_L$-only ablation settle this directly.

Why not just fine-tune a stronger pretrained or geospatial encoder on the few demonstrations? A better encoder might close the label gap with no new objective. However, How to Efficiently Adapt Foundation Models to Remote Sensing for Object Detection? reports that lightweight adaptation of web models fails on overhead geometry. And InfoGeo needs explicit object-centric structure for its DINOv2-based matcher to survive cross-domain shift. Neither result is about navigation, so we test this alternative instead of assuming it fails. We swap in a stronger frozen egocentric encoder for both the naive baseline and our method. If our gain disappears with the stronger encoder, the contribution reduces to "use a better backbone."

Why not just add shortest-path trajectories as extra demonstrations? CityNav's 85.1% versus 56.6% shows that shortest-path actions are a poor substitute for human ones. That comparison, however, was of replacement, not of mixing cheap flights in with a few human flights. The action-augmentation baseline measures mixing at every budget. Our claim fails if it matches observation-only use of the same flights.

Why not just fill in the missing map with a generative model? Cross-view generators such as CrossViewDiff can synthesize one view from another. Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classification notes that imputed content can introduce errors that propagate downstream, and a generator adds per-step cost in a control loop. More basically, imputation still needs a policy that understands the map, and it does nothing for label efficiency. We therefore treat it as a robustness-only comparison: a predicted-map variant in which our own decoder's output is fed back as the map input. We do not train a separate diffusion model.

Proposed Experiments

All runs use CityNav's test-unseen split with success rate as the primary metric. We also report the benchmark's other standard navigation metrics where its evaluation code provides them.

Label efficiency. We plot a success-versus-demonstration-budget curve for every method with the map present. The decisive check: at 5% and 10% of the human flights, our method should recover most of the gap between the low-budget naive baseline and the full-data AerialVLN+GSM. Dropout-alone and StreetAir-style alignment should recover visibly less.

Robustness to map loss and conflict. We evaluate four test conditions:

  • map present throughout;
  • map absent for the whole episode;
  • map dropped at a random mid-episode step;
  • map present but stale, with its landmark layer shifted by a fixed offset or with obstacles removed.

The stale-map case tests the collapse toward a single modality described in the audio-visual conflict study. Here a successful method should lose less than the naive baseline, which we expect to follow the corrupted map. Alongside success, we measure how strongly each method's actions depend on the map by computing the change in action distribution under map perturbation.

Mechanism checks. - Map-prediction accuracy on held-out areas should correlate with map-absent success across seeds and settings. If it does not, the decoder is learning something the policy does not use. - The comparison of shortest-path observations against human-flight observations tells us whether cheap coverage suffices. - The orthophoto variant tells us whether the benefit survives replacing the clean symbolic map with raw top-down imagery, where the remote-sensing gap is real.

A negative result in that last condition would still be informative. It would locate the method's limit at the symbolic-map setting.