TL;DR
NereidVLA is a two-stage vision-language-action model that bridges the gap between marine spatial understanding and 3D robot navigation. By first pretraining on two million spatial reasoning examples and then adapting to expert trajectories, it achieves a 78.75% success rate in real-world underwater trials—nearly doubling the performance of previous state-of-the-art models without any real-world fine-tuning.
What Problem Does This Paper Address?
Navigating underwater is a unique challenge for robotics. Between the 3D complexity of the environment, poor visibility due to turbidity, and the way light scatters in water, standard navigation algorithms often struggle.
While the field of Vision-Language-Action (VLA) models has seen massive success on land, the underwater domain lacks the massive, paired "image-to-robot-motion" datasets required to train these models from scratch. We have plenty of marine imagery with labels (e.g., "this is a coral," "the fish is behind the reef"), but very little data showing how a robot should actually move in response to those scenes. The authors of NereidVLA ask: Can we transfer general marine spatial knowledge into actionable 3D navigation?
Methods
The researchers developed a two-stage learning framework to solve the data scarcity problem:
- Stage 1: Marine Spatial Learning (Sea2Know): Instead of starting with motion, they first taught a model to "understand" the ocean. They trained a model called Sea2Know (based on the Qwen3-VL-8B backbone) on approximately two million examples. These examples weren't about moving; they were about answering spatial questions: "Is the anchor to the left of the diver?" or "Is the cuttlefish in the foreground?" This established a deep "spatial prior" for underwater scenes.
- Stage 2: Navigation Adaptation (NereidVLA): They then initialized NereidVLA with the weights from Sea2Know and introduced motion. Using a simulation environment called MarineGym, they collected ~100,000 navigation samples.
- Residual-Aware Motion Tokenization: To turn visual inputs into movement, the model predicts "motion tokens." Rather than predicting raw coordinates, it selects from a codebook of 1,024 discrete 3D displacements. Crucially, it uses a "residual-aware" approach where each predicted step accounts for the quantization errors of the previous step, leading to much smoother trajectories.
The final system takes a 4-frame RGB history and a language instruction (e.g., "Follow the pipe toward the exit") and predicts a 5-second local 3D trajectory, which is then executed by a low-level tracking controller.
Datasets
- Spatial Pretraining: Combined data from NAUTILUS and USI-Brain, totaling ~2 million marine spatial-understanding examples.
- Navigation Training: 5,117 expert demonstrations across 573 simulated scenes in MarineGym, covering obstacle avoidance, target search, pipeline navigation, and cave navigation.
- New Benchmark: The authors introduced a six-dimensional Underwater Navigation Understanding Benchmark consisting of 3,646 QA items. It tests models on Target Grounding (TGA), Spatial Relations (SRA), Passage Structure (TSA), Traversability (TPA), Temporal Understanding (TUA), and Local Action Decisions (LADA).
Key Insights and Ideas
- Spatial Knowledge is a Prerequisite for Motion: The most striking finding is that learning "where things are" is more important than learning "how to move" initially. Models that skipped the spatial pretraining stage performed significantly worse at navigation.
- Temporal Understanding is the "Aha" Moment: In the benchmark tests, Sea2Know showed its biggest gains in Temporal Understanding (+28.97 points). In the flickering, murky underwater world, being able to track how a target's appearance changes over time is the difference between a successful mission and a collision.
- Sim-to-Real Transfer is Possible: Because the model learned general spatial principles (like depth and occlusion) rather than just memorizing simulation textures, it transferred remarkably well to a physical BlueROVHeavy robot in a laboratory pool without any additional training.
Key Results
- Benchmark Performance: Sea2Know (8B) outperformed its backbone across all six dimensions of the new benchmark, particularly in spatial relations and local action decisions.
- Simulation Success: Under matched settings, the marine spatial pretraining increased the navigation success rate from 72.4% to 84.4%.
- Real-World Trials: In zero-shot laboratory-pool tests, NereidVLA achieved a 78.75% task success rate. For comparison, the previous state-of-the-art model, InternVLA-N1, only managed 47.50%.
Limitations and Future Work
- Discrete Action Space: The use of a fixed codebook for motion tokens, while robust, may limit the robot's ability to perform extremely fine-grained or high-speed maneuvers that require continuous control.
- Environmental Diversity: While the model handles pool environments well, the "real" ocean presents even more extreme turbidity and dynamic obstacles (like schools of fish) that may require even more diverse training data.
- Future Directions: The authors suggest that scaling the model size further and incorporating more diverse real-world marine observations could lead to a truly "foundation" model for underwater robotics.
Summary updated