MDE-AgriVLN: Agricultural Vision-and-Language Navigation with Monocular Depth Estimation

Xiaobei Zhao, Xingqi Lyu, Xin Chen, Xiang LiarXiv 2025 · DOI · arXiv

Abstract

Autonomous navigation in agricultural environments often relies on monocular vision, which lacks the inherent depth perception of binocular systems. This paper introduces MDE-AgriVLN, a framework that integrates monocular depth estimation (MDE) into Vision-Language Models (VLMs) to bridge the spatial perception gap. By providing metric depth features, MDE-AgriVLN improves the success rate of following natural language instructions by nearly 40% over existing baselines on the Agriculture-to-Agriculture (A2A) benchmark.

Introduction

While robots are increasingly deployed in agriculture for monitoring and weeding, they typically operate on fixed paths or via manual control. Vision-and-Language Navigation (VLN) offers a more flexible alternative, allowing robots to follow natural language commands such as "Go to the blue box with plants at the end of the road." However, monocular systems frequently suffer from "early stopping"—halting several meters before a target due to poor distance estimation. This research aims to provide these robots with a "spatial sense" using zero-shot metric depth estimation, avoiding the need for expensive LiDAR sensors.

Baseline performance on a sample episode illustrating the 'early stopping' problem where the robot halts 3.63m from the target.

Methodology

The MDE-AgriVLN framework augments the standard navigation pipeline with a Monocular Depth Estimation module.

  1. Monocular Depth Estimation: The system processes RGB images through Depth Pro, a zero-shot metric depth estimator. It generates a Depth Matrix (a 16x9 grid of absolute distances in meters) and a Depth Map (a color-coded visualization of the 3D structure).
  2. Multimodal Reasoning: These features are integrated into a Vision-Language Model (Qwen2.5-VL-32B) paired with a reasoning model (DeepSeek-R1-32B).
  3. Action Selection: The model generates a chain-of-thought reasoning process based on the instruction, RGB image, and depth data to select actions like FORWARD or STOP based on calculated proximity to targets.
MDE-AgriVLN performance on the same sample episode, showing successful navigation (1.71m error) by integrating depth input and reasoning.

Evaluation and Results

The model was evaluated on the Agriculture-to-Agriculture (A2A) benchmark, covering 1,560 episodes across diverse scenes including farms, greenhouses, and forests.

Key Metrics

  • Success Rate (SR): MDE-AgriVLN achieved a success rate of 0.32, a 39% relative improvement over the 0.23 baseline.
  • Navigation Error (NE): The average stopping distance from the target was reduced from 4.43m to 4.08m.

Insights

  • Metric Depth Importance: Absolute metric depth was found to be significantly more effective than relative depth for satisfying specific "stop" conditions in instructions.
  • Environment Complexity: The impact of MDE was most pronounced in chaotic environments like forests, where visual size is a poor proxy for distance due to irregular tree sizes.
  • Alignment Burden: Interestingly, providing both a depth matrix and a depth map did not always outperform the matrix alone, suggesting that redundant spatial data can sometimes distract the VLM.

Conclusion and Future Work

MDE-AgriVLN demonstrates that integrating monocular depth estimation significantly enhances the spatial awareness of agricultural robots. Future work will address current limitations, including spatial misalignment between RGB and depth features, computational latency (currently 0.2s–0.8s per step), and robustness to ambiguous human instructions.

Summary updated

BibTeX
@misc{zhao2025mde,
  author = {Xiaobei Zhao and Xingqi Lyu and Xin Chen and Xiang Li},
  title = {{MDE-AgriVLN: Agricultural Vision-and-Language Navigation with Monocular Depth Estimation}},
  year = {2025},
  eprint = {2512.03958},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.03958},
}
Figure