NereidVLA: Learning to Understand and Navigate the Underwater World

Chenjie Zhao, Zhiliang Zhu, Quan Chen, Yuquan Xue, Xingxue Shi, Puyue Wang, Zhiqiang Zhao, Kangye Zheng, Boyang Xu, Mingwei Lin, Dejun LiarXiv 2026 · arXiv

TL;DR

NereidVLA is a two-stage vision-language-action model that bridges the gap between marine spatial understanding and 3D robot navigation. By first pretraining on two million spatial reasoning examples and then adapting to expert trajectories, it achieves a 78.75% success rate in real-world underwater trials—nearly doubling the performance of previous state-of-the-art models without any real-world fine-tuning.

What Problem Does This Paper Address?

Navigating underwater is a unique challenge for robotics. Between the 3D complexity of the environment, poor visibility due to turbidity, and the way light scatters in water, standard navigation algorithms often struggle.

While the field of Vision-Language-Action (VLA) models has seen massive success on land, the underwater domain lacks the massive, paired "image-to-robot-motion" datasets required to train these models from scratch. We have plenty of marine imagery with labels (e.g., "this is a coral," "the fish is behind the reef"), but very little data showing how a robot should actually move in response to those scenes. The authors of NereidVLA ask: Can we transfer general marine spatial knowledge into actionable 3D navigation?

Methods

The researchers developed a two-stage learning framework to solve the data scarcity problem:

  1. Stage 1: Marine Spatial Learning (Sea2Know): Instead of starting with motion, they first taught a model to "understand" the ocean. They trained a model called Sea2Know (based on the Qwen3-VL-8B backbone) on approximately two million examples. These examples weren't about moving; they were about answering spatial questions: "Is the anchor to the left of the diver?" or "Is the cuttlefish in the foreground?" This established a deep "spatial prior" for underwater scenes.
  2. Stage 2: Navigation Adaptation (NereidVLA): They then initialized NereidVLA with the weights from Sea2Know and introduced motion. Using a simulation environment called MarineGym, they collected ~100,000 navigation samples.
  3. Residual-Aware Motion Tokenization: To turn visual inputs into movement, the model predicts "motion tokens." Rather than predicting raw coordinates, it selects from a codebook of 1,024 discrete 3D displacements. Crucially, it uses a "residual-aware" approach where each predicted step accounts for the quantization errors of the previous step, leading to much smoother trajectories.
Figure 1. Overview of the two-stage learning framework for NereidVLA. Marine spatial learning produces Sea2Know, which initializes the navigation model. Expert demonstrations executed under simulated vehicle dynamics supervise local 3D trajectory prediction, followed by closed-loop tracking with a low-level feedback controller.

The final system takes a 4-frame RGB history and a language instruction (e.g., "Follow the pipe toward the exit") and predicts a 5-second local 3D trajectory, which is then executed by a low-level tracking controller.

Datasets

  • Spatial Pretraining: Combined data from NAUTILUS and USI-Brain, totaling ~2 million marine spatial-understanding examples.
  • Navigation Training: 5,117 expert demonstrations across 573 simulated scenes in MarineGym, covering obstacle avoidance, target search, pipeline navigation, and cave navigation.
  • New Benchmark: The authors introduced a six-dimensional Underwater Navigation Understanding Benchmark consisting of 3,646 QA items. It tests models on Target Grounding (TGA), Spatial Relations (SRA), Passage Structure (TSA), Traversability (TPA), Temporal Understanding (TUA), and Local Action Decisions (LADA).
Figure 3. Underwater Navigation Understanding Benchmark. The six panels show representative examples and question proportions for target grounding, spatial relations, passage structure, traversability, temporal understanding, and local action decisions. Each example pairs visual observations with a question, answer options, and the reference answer.

Key Insights and Ideas

  • Spatial Knowledge is a Prerequisite for Motion: The most striking finding is that learning "where things are" is more important than learning "how to move" initially. Models that skipped the spatial pretraining stage performed significantly worse at navigation.
  • Temporal Understanding is the "Aha" Moment: In the benchmark tests, Sea2Know showed its biggest gains in Temporal Understanding (+28.97 points). In the flickering, murky underwater world, being able to track how a target's appearance changes over time is the difference between a successful mission and a collision.
  • Sim-to-Real Transfer is Possible: Because the model learned general spatial principles (like depth and occlusion) rather than just memorizing simulation textures, it transferred remarkably well to a physical BlueROVHeavy robot in a laboratory pool without any additional training.

Key Results

  • Benchmark Performance: Sea2Know (8B) outperformed its backbone across all six dimensions of the new benchmark, particularly in spatial relations and local action decisions.
  • Simulation Success: Under matched settings, the marine spatial pretraining increased the navigation success rate from 72.4% to 84.4%.
  • Real-World Trials: In zero-shot laboratory-pool tests, NereidVLA achieved a 78.75% task success rate. For comparison, the previous state-of-the-art model, InternVLA-N1, only managed 47.50%.
Figure 4. Closed-loop navigation examples. (a) Target search and in-pipe navigation in simulation, showing successive frames, onboard views, and predicted and executed trajectories. (b) Real-robot goal approach with obstacles and narrow-passage avoidance, shown through external and onboard views. Time labels indicate stages of each execution.

Limitations and Future Work

  • Discrete Action Space: The use of a fixed codebook for motion tokens, while robust, may limit the robot's ability to perform extremely fine-grained or high-speed maneuvers that require continuous control.
  • Environmental Diversity: While the model handles pool environments well, the "real" ocean presents even more extreme turbidity and dynamic obstacles (like schools of fish) that may require even more diverse training data.
  • Future Directions: The authors suggest that scaling the model size further and incorporating more diverse real-world marine observations could lead to a truly "foundation" model for underwater robotics.

Summary updated

Abstract

Underwater navigation requires translating visual spatial cues into motion, yet marine vision-language datasets rarely include paired robot trajectories. We present NereidVLA, a vision-language-action model that transfers marine spatial knowledge to language-conditioned 3D navigation. We first train Sea2Know on approximately two million marine examples to learn target grounding, spatial relations, and scene structure without paired robot-motion supervision. We then adapt it using expert trajectories executed under simulated underwater vehicle dynamics. Given RGB histories and a language instruction, NereidVLA predicts discrete motion tokens that decode into local 3D trajectories for closed-loop tracking. Evaluation combines an independent six-dimensional underwater navigation understanding benchmark with offline prediction and closed-loop navigation. Under matched navigation training and execution settings, marine spatial pretraining increases simulation success from 72.4% to 84.4%. Without fine-tuning on the real test environment, NereidVLA achieves 78.75% success in laboratory-pool robot trials, compared with 47.50% for InternVLA-N1. These results show that marine spatial supervision can support downstream motion learning and physical underwater navigation.
BibTeX
@misc{zhao2026nereidvla,
  author = {Chenjie Zhao and Zhiliang Zhu and Quan Chen and Yuquan Xue and Xingxue Shi and Puyue Wang and Zhiqiang Zhao and Kangye Zheng and Boyang Xu and Mingwei Lin and Dejun Li},
  title = {{NereidVLA: Learning to Understand and Navigate the Underwater World}},
  year = {2026},
  eprint = {2611.24864},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24864},
}
Figure