WayTrace: A Map-to-Language-to-Trace Hierarchy for Embodied Urban Navigation

Manthan Patel, Jonas Frey, Pascal Roth, Marco Hutter, Cesar CadenaarXiv 2026 · arXiv

TL;DR

WayTrace is a hierarchical navigation system that enables robots to follow high-level web-map directions (like Google Maps) by translating them into atomic language commands and then into visual "traces" in the camera view. By using language as an interface instead of precise metric coordinates, WayTrace achieves remarkable robustness to GNSS noise and successfully completes long-range missions of up to 1.37 km in complex real-world urban environments.

What Problem Does This Paper Address?

Autonomous urban navigation typically falls into two camps: point-goal navigation, which follows a sequence of GNSS waypoints, and Vision-Language Navigation (VLN), which follows natural language instructions.

Both have significant drawbacks. Point-goal methods are at the mercy of GNSS accuracy; in "urban canyons," errors of several meters are common, often leading a robot to attempt to drive through walls or off sidewalks. Conversely, current VLN models are often "brittle," struggling to chain together the many short instructions required for a kilometer-long journey. Furthermore, neither approach inherently respects social norms, like staying on sidewalks or using zebra crossings, unless explicitly (and often painstakingly) programmed to do so.

Methods

The authors propose a four-level hierarchy called WayTrace that connects high-level maps to low-level motor control:

  1. MAP: A web-map service (like Google Routes) provides standard turn-by-turn directions.
  2. LANGUAGE (The Orchestrator): An off-the-shelf Vision-Language Model (VLM), specifically Qwen3.8-Flash-Next, acts as a high-level planner. It receives the directions, a satellite map crop, and a history of camera images. It then issues a single "atomic" command, such as "Go left onto the crosswalk to the far end kerb."
  3. TRACE (The Predictor): An onboard, language-conditioned model (RynnBrain-9B) takes the atomic command and the current camera view to draw a 10-point "trace" (a path) directly in the image space. Crucially, it uses a temporal history of past frames to understand its progress (e.g., knowing a turn is complete).
  4. MOTION (The Policy): The 2D trace is lifted into 3D using depth information. A Reinforcement Learning (RL) policy then tracks a "carrot" goal 4.5 meters ahead along this path, handling local obstacle avoidance and locomotion.
Overview of the WayTrace hierarchy showing the flow from map directions to orchestrator reasoning, trace prediction, and RL-based motion control.

Datasets

To train the Trace Predictor, the authors compiled a massive dataset of 1.19 million samples:

  • Reconstructed Walks (288k samples): Derived from the Mapillary dataset. They used 3D reconstruction (VGGT-Ω) to project the actual paths walked by humans back into their camera views, labeled with VLM-generated commands.
  • Intent-Conditioned Traces (863k samples): To teach the model to follow specific commands at junctions (e.g., "turn left" vs. "go straight"), they used a VLM "teacher" to generate multiple valid commands for a single scene and a teacher model to draw the corresponding traces.
  • Simulation Benchmark: A high-fidelity reconstruction of Zurich’s old town in NVIDIA Isaac Sim 5.0, featuring 123 zebra crossings and 880 shops for realistic testing.
The training data pipeline, illustrating how reconstructed human walks and intent-conditioned VLM-generated traces are used to build the 1.19M sample dataset.

Key Insights and Ideas

  • Language as a Robust Abstraction: By using language commands (like "cross the road") as the interface between the map and the robot, the system becomes decoupled from precise metric coordinates. This makes it inherently resilient to GNSS drift.
  • Temporal History Matters: A "blind" trace predictor that only sees the current frame often gets stuck in loops (e.g., continuing to turn after a turn is finished). Including a history of past frames allows the model to recognize the completion of a maneuver.
  • Intent-Conditioned Training: Training on multiple possible paths from the same junction is the "secret sauce" that prevents the model from simply predicting the most likely path and instead forces it to follow the specific language command.

Key Results

  • GNSS Resilience: In simulation, WayTrace maintained a 63% success rate even with a 7-meter RMS GNSS error. In contrast, standard point-goal baselines dropped below 25% success and committed ten times more sidewalk/crosswalk violations.
  • Real-World Success: Deployed on a Unitree B2W wheeled-legged robot, WayTrace completed multiple missions in Zurich. This included a 1.37 km mission through rural trails and a university campus, and a 426-meter mission through a busy city center that involved navigating stairs and construction sites with 100% autonomy.
  • Superior Trace Prediction: On the "Human-Test" benchmark, WayTrace achieved a NaviTrace score of 83.2, vastly outperforming existing VLN baselines like VAMOS (56.6) and Foresight (50.5).
Success rates and violations under varying levels of GNSS noise, showing WayTrace's superior robustness compared to point-goal baselines.

Limitations and Future Work

  • Dynamic Obstacles: The current system does not explicitly reason about traffic. It cannot yet wait for a green light or yield to an approaching tram (though the low-level RL policy does avoid pedestrians).
  • Latency: The VLM orchestrator takes ~1.3 seconds per decision. While the robot moves continuously, this latency limits the maximum safe speed (currently capped at 1.3 m/s).
  • Visibility: The system struggles if a command refers to a landmark that is no longer visible (e.g., a shop the robot has just passed).
  • Future Directions: The authors suggest fine-tuning the orchestrator VLM on "verified" decisions and integrating multi-camera views to provide better context for maneuvers.

For more details, visit the project page.

Summary updated

Abstract

Web services provide walking directions for nearly every city, but following them reliably over long routes remains challenging for autonomous robots. Point-goal navigation policies rely heavily on accurate GNSS, while most vision-language navigation models struggle to execute even short instructions outdoors. We present WayTrace, a hierarchical system that connects web-map directions to robot motion through language. An off-the-shelf orchestrator vision-language model reads the turn-by-turn directions, a satellite map and recent camera images, and issues one atomic language command at a time, such as "cross the road to the left". An onboard trace predictor, trained on over one million automatically labelled samples from street-level images, draws the path for this command in the image, and a reinforcement learning policy tracks it. In long-range simulated missions, WayTrace remains robust to GNSS noise and rarely leaves crosswalks or sidewalks, while point-goal baselines degrade sharply with about ten times more violations. Deployed on a wheeled-legged robot, WayTrace completes real-world missions of up to 1.37 km at about 1.1 m/s through a busy city centre, rural trails and a university campus, handling crosswalks, staircases and construction sites. Project page: https://way-trace.github.io
BibTeX
@misc{patel2026waytrace,
  author = {Manthan Patel and Jonas Frey and Pascal Roth and Marco Hutter and Cesar Cadena},
  title = {{WayTrace: A Map-to-Language-to-Trace Hierarchy for Embodied Urban Navigation}},
  year = {2026},
  eprint = {2611.05129},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05129},
}
Figure