TL;DR
DeRep is a novel representation-learning method for robot navigation that combines standard depth reconstruction with a task-specific "reachability" objective. By explicitly training an encoder to predict local free-space connectivity, the authors achieve significantly higher success rates and better robustness to sensor noise in complex, cluttered environments compared to traditional methods.
What Problem Does This Paper Address?
End-to-end navigation—where a robot maps raw sensor data (like depth images) directly to motor commands—is a powerful paradigm. However, it often relies on a "black box" encoder to compress high-dimensional images into a compact latent representation.
The challenge lies in what this representation should actually capture. Standard approaches either train the encoder from scratch alongside the navigation policy (which can be unstable) or use "autoencoding" to reconstruct the original image. While reconstruction preserves visual detail, it doesn't explicitly prioritize the information a robot needs most: "Where can I safely move?" A representation might perfectly reconstruct a wall but fail to clearly signal a narrow, flyable gap. This paper investigates whether explicitly supervising local "reachability" provides a better foundation for safe, goal-directed flight.
Methods
The authors propose DeRep, a joint pretraining framework for depth encoders. The core idea is to train a CNN encoder using two simultaneous decoders that are discarded after training:
- Depth Reconstruction Decoder: A standard autoencoder branch that forces the latent representation to retain fine-grained geometric details of the scene.
- Local BEV Reachability Decoder: This is the "secret sauce." It predicts an ego-centric Bird’s-Eye-View (BEV) map of the space immediately around the robot. Specifically, it predicts which grid cells are "reachable" (connected to the robot's current position) and the shortest-path distance to each cell.
By combining these, the encoder learns both the "low-level" geometry and the "high-level" topological connectivity of the environment. Crucially, once pretrained on a large dataset, the encoder is frozen. This allows for a fair comparison across different downstream learners, including linear probes, imitation learning (via flow-matching), and reinforcement learning (PPO).
Datasets
- Pretraining: The encoders were trained on approximately 1.15 million single-frame depth observations rendered from the Gibson dataset within the Habitat simulator. This provides a diverse set of realistic indoor geometries.
- Evaluation: Downstream policies were trained and tested in Isaac Sim using synthetic obstacle layouts. These included "In-Distribution" (ID) scenes, "Out-of-Distribution" (OOD) geometries (like tilted or concave obstacles), and scenes with various levels of sensor noise (Gaussian noise, pixel dropout, and quantization).
Key Insights and Ideas
- Reachability as a Topological Map: Reachability supervision acts as a powerful inductive bias. It forces the latent space to understand the "navigability" of a scene. This is more useful for a policy than just knowing where pixels are; it's about knowing which gaps are traversable.
- The Best of Both Worlds: While reachability alone is strong, the joint objective (DeRep) performs best in Reinforcement Learning. Reconstruction likely preserves subtle geometric cues that help the policy fine-tune its motion, while reachability provides the "big picture" of safe paths.
- Robustness to "Dirty" Data: One of the most striking findings is how reachability-based features handle sensor noise. Because the reachability objective focuses on structural connectivity rather than pixel-perfect reconstruction, the resulting features are much more resilient when the input depth image is corrupted by noise or low resolution.
- Zero-Shot Transfer: Because the representation is learned on diverse data and captures fundamental navigation concepts, the resulting policies can be moved from simulation to the real world without any fine-tuning.
Key Results
The authors compared DeRep against several baselines, including training from scratch, standard Autoencoders (AE), and Variational Autoencoders (VAE).
- Reinforcement Learning (PPO): In-distribution, DeRep achieved an 84.0% success rate, significantly outperforming the reconstruction-only AE (67.3%) and training from scratch (33.5%).
- Generalization: Under OOD conditions with tilted obstacles, DeRep maintained a 77.3% success rate, whereas the AE baseline dropped to 59.3%.
- Noise Robustness: At medium noise levels, DeRep outperformed the AE baseline by 19.0 percentage points in success rate.
- Real-World Deployment: The authors successfully deployed a PPO policy with a frozen DeRep encoder on a physical quadrotor (Jetson Orin NX + RealSense D435i). The drone navigated cluttered indoor environments and even avoided a walking person in a zero-shot manner.
Limitations and Future Work
- Temporal Information: Currently, DeRep processes only a single depth frame at a time. It does not have a "memory" of past observations, which limits its performance in highly dynamic environments or when obstacles are temporarily occluded.
- Static Environments: The reachability targets are generated from static navigation meshes. Extending this to handle moving obstacles (like other robots or people) in the pretraining phase is a logical next step.
- Task Specificity: While reachability is great for "point A to point B" navigation, different representations might be needed for more complex semantic tasks, like "find the kitchen."
Summary updated