DeRep: Learning Depth Representations for End-to-End Navigation in Cluttered Environments

Hanyu Jin, Kenji ShimadaarXiv 2026 · arXiv

TL;DR

DeRep is a novel representation-learning method for robot navigation that combines standard depth reconstruction with a task-specific "reachability" objective. By explicitly training an encoder to predict local free-space connectivity, the authors achieve significantly higher success rates and better robustness to sensor noise in complex, cluttered environments compared to traditional methods.

What Problem Does This Paper Address?

End-to-end navigation—where a robot maps raw sensor data (like depth images) directly to motor commands—is a powerful paradigm. However, it often relies on a "black box" encoder to compress high-dimensional images into a compact latent representation.

The challenge lies in what this representation should actually capture. Standard approaches either train the encoder from scratch alongside the navigation policy (which can be unstable) or use "autoencoding" to reconstruct the original image. While reconstruction preserves visual detail, it doesn't explicitly prioritize the information a robot needs most: "Where can I safely move?" A representation might perfectly reconstruct a wall but fail to clearly signal a narrow, flyable gap. This paper investigates whether explicitly supervising local "reachability" provides a better foundation for safe, goal-directed flight.

Methods

The authors propose DeRep, a joint pretraining framework for depth encoders. The core idea is to train a CNN encoder using two simultaneous decoders that are discarded after training:

  1. Depth Reconstruction Decoder: A standard autoencoder branch that forces the latent representation to retain fine-grained geometric details of the scene.
  2. Local BEV Reachability Decoder: This is the "secret sauce." It predicts an ego-centric Bird’s-Eye-View (BEV) map of the space immediately around the robot. Specifically, it predicts which grid cells are "reachable" (connected to the robot's current position) and the shortest-path distance to each cell.
Fig. 2. Overview of the proposed framework. (a) Representation pretraining: the depth encoder is jointly trained with a local BEV reachability decoder and a depth reconstruction decoder. (b) Downstream evaluation: after pretraining, the decoders are discarded and the frozen encoder is evaluated with a linear policy probe, flow-matching imitation learning, and PPO.

By combining these, the encoder learns both the "low-level" geometry and the "high-level" topological connectivity of the environment. Crucially, once pretrained on a large dataset, the encoder is frozen. This allows for a fair comparison across different downstream learners, including linear probes, imitation learning (via flow-matching), and reinforcement learning (PPO).

Datasets

  • Pretraining: The encoders were trained on approximately 1.15 million single-frame depth observations rendered from the Gibson dataset within the Habitat simulator. This provides a diverse set of realistic indoor geometries.
  • Evaluation: Downstream policies were trained and tested in Isaac Sim using synthetic obstacle layouts. These included "In-Distribution" (ID) scenes, "Out-of-Distribution" (OOD) geometries (like tilted or concave obstacles), and scenes with various levels of sensor noise (Gaussian noise, pixel dropout, and quantization).

Key Insights and Ideas

  • Reachability as a Topological Map: Reachability supervision acts as a powerful inductive bias. It forces the latent space to understand the "navigability" of a scene. This is more useful for a policy than just knowing where pixels are; it's about knowing which gaps are traversable.
  • The Best of Both Worlds: While reachability alone is strong, the joint objective (DeRep) performs best in Reinforcement Learning. Reconstruction likely preserves subtle geometric cues that help the policy fine-tune its motion, while reachability provides the "big picture" of safe paths.
  • Robustness to "Dirty" Data: One of the most striking findings is how reachability-based features handle sensor noise. Because the reachability objective focuses on structural connectivity rather than pixel-perfect reconstruction, the resulting features are much more resilient when the input depth image is corrupted by noise or low resolution.
  • Zero-Shot Transfer: Because the representation is learned on diverse data and captures fundamental navigation concepts, the resulting policies can be moved from simulation to the real world without any fine-tuning.

Key Results

The authors compared DeRep against several baselines, including training from scratch, standard Autoencoders (AE), and Variational Autoencoders (VAE).

  • Reinforcement Learning (PPO): In-distribution, DeRep achieved an 84.0% success rate, significantly outperforming the reconstruction-only AE (67.3%) and training from scratch (33.5%).
  • Generalization: Under OOD conditions with tilted obstacles, DeRep maintained a 77.3% success rate, whereas the AE baseline dropped to 59.3%.
  • Noise Robustness: At medium noise levels, DeRep outperformed the AE baseline by 19.0 percentage points in success rate.
  • Real-World Deployment: The authors successfully deployed a PPO policy with a frozen DeRep encoder on a physical quadrotor (Jetson Orin NX + RealSense D435i). The drone navigated cluttered indoor environments and even avoided a walking person in a zero-shot manner.
Real-world deployment of the DeRep policy on a physical quadrotor navigating a cluttered room.

Limitations and Future Work

  • Temporal Information: Currently, DeRep processes only a single depth frame at a time. It does not have a "memory" of past observations, which limits its performance in highly dynamic environments or when obstacles are temporarily occluded.
  • Static Environments: The reachability targets are generated from static navigation meshes. Extending this to handle moving obstacles (like other robots or people) in the pretraining phase is a logical next step.
  • Task Specificity: While reachability is great for "point A to point B" navigation, different representations might be needed for more complex semantic tasks, like "find the kitchen."

Summary updated

Abstract

End-to-end navigation in cluttered environments relies on compact visual representations to support collision avoidance and goal-directed motion. Existing methods typically learn these representations jointly with the policy or through observation reconstruction, which does not explicitly specify what information the representation should preserve. Moreover, they are usually evaluated with a single downstream policy learner, although the benefit of a representation may depend on how the policy is learned. It therefore remains unclear how different representation-learning objectives affect downstream navigation performance and robustness. This work proposes DeRep, a navigation-oriented representation-learning method that combines local bird's-eye-view reachability supervision with depth reconstruction. To compare representation objectives under controlled conditions, we pretrain encoders with a common architecture and freeze them during downstream learning. We evaluate the representations using a linear policy probe, flow-matching imitation learning, and reinforcement learning with PPO. DeRep achieves the lowest linear policy probe prediction error and outperforms reconstruction alone across all six tested PPO conditions. In imitation learning, both reachability-based objectives achieve higher mean success rates than reconstruction alone, with similar mean success rates between the reachability-only and joint objectives. These results demonstrate consistent benefits of incorporating reachability supervision into depth representation pretraining across different downstream learning methods and evaluation conditions. We further deploy the PPO policy with the frozen DeRep encoder in Gazebo and on a physical quadrotor without fine-tuning, demonstrating its capability for zero-shot sim-to-sim and sim-to-real transfer.
BibTeX
@misc{jin2026derep,
  author = {Hanyu Jin and Kenji Shimada},
  title = {{DeRep: Learning Depth Representations for End-to-End Navigation in Cluttered Environments}},
  year = {2026},
  eprint = {2611.13499},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13499},
}
Figure