SUM-AgriVLN: Spatial Understanding Memory for Agricultural Vision-and-Language Navigation

Xiaobei Zhao, Xingqi Lyu, Xiang LiarXiv 2025 · DOI · arXiv

Introduction

Vision-and-Language Navigation (VLN) enables robots to execute complex tasks based on natural language instructions. While significant progress has been made in indoor and urban settings, agricultural environments present unique challenges. Specifically, agricultural robots often utilize low-mounted cameras (approximately 0.38m) to monitor crops and terrain. This low perspective results in "tunnel vision," where the robot lacks the global spatial context necessary for long-range planning. Furthermore, standard VLN models treat each task as an independent episode, effectively suffering from "amnesia" regarding previously traversed paths.

The SUM-AgriVLN Framework

To address these limitations, we introduce SUM-AgriVLN, a framework that incorporates a Spatial Understanding Memory (SUM) module. This module allows robots to "remember" the 3D layout of a farm from previous runs, providing the necessary global context to improve navigation accuracy.

Comparison between standard agricultural VLN methods and the proposed SUM-AgriVLN, highlighting the role of spatial memory in accurate navigation.

The SUM module operates in two distinct phases:

  1. Spatial Reconstruction: Utilizing a Visual Geometry Grounded Transformer (VGGT), the system performs 3D reconstruction from sampled camera frames, converting 2D image sequences into a 3D mesh representation.
  2. Memory Storage and Recall: The 3D reconstruction is rendered using the trimesh tool into two specific 2D perspectives—Frontal (0–5°) and Oblique (40–50°). These views are stored in a Spatial Memory Bank. During navigation, the robot recalls these memories to reason about its current position relative to the global map.

Experimental Results

The proposed method was evaluated on the A2A (Agricultural-to-Action) benchmark, which comprises 1,560 episodes across diverse agricultural settings, including greenhouses, forests, and gardens.

SUM-AgriVLN achieved a state-of-the-art Success Rate (SR) of 0.54, a notable improvement over the previous baseline of 0.47. In low-complexity scenarios involving shorter paths, the success rate reached as high as 0.66. While the Navigation Error (NE) was slightly higher than some specialized models (2.93m vs 2.60m), the significant gain in overall success rate demonstrates the effectiveness of spatial memory in reaching target destinations.

Key Insights

  • The Value of Oblique Perspectives: Research indicates that oblique (angled) views often provide better navigation cues than frontal views in dense agricultural environments, as they offer a "top-down" perspective that avoids occlusion by tall crops.
  • Macro Perception: Spatial memory enables "macro perception," allowing the robot to judge distances more accurately. Unlike baseline models that might stop prematurely due to 2D visual scaling, SUM-AgriVLN utilizes its memory of the path's actual length to ensure precise positioning.

Limitations and Future Work

Despite its success, SUM-AgriVLN faces several hurdles. The current 3D reconstruction process assumes static scenes and struggles with dynamic objects like moving workers. Additionally, converting 3D memory into 2D RGB images results in some information loss. Future research will focus on real-time spatial understanding and more robust 3D embeddings to handle dynamic agricultural environments.

Summary updated

BibTeX
@misc{zhao2025sum,
  author = {Xiaobei Zhao and Xingqi Lyu and Xiang Li},
  title = {{SUM-AgriVLN: Spatial Understanding Memory for Agricultural Vision-and-Language Navigation}},
  year = {2025},
  eprint = {2510.14357},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.14357},
}
Figure