Abstract
Autonomous navigation in agricultural environments often relies on monocular vision, which lacks the inherent depth perception of binocular systems. This paper introduces MDE-AgriVLN, a framework that integrates monocular depth estimation (MDE) into Vision-Language Models (VLMs) to bridge the spatial perception gap. By providing metric depth features, MDE-AgriVLN improves the success rate of following natural language instructions by nearly 40% over existing baselines on the Agriculture-to-Agriculture (A2A) benchmark.
Introduction
While robots are increasingly deployed in agriculture for monitoring and weeding, they typically operate on fixed paths or via manual control. Vision-and-Language Navigation (VLN) offers a more flexible alternative, allowing robots to follow natural language commands such as "Go to the blue box with plants at the end of the road." However, monocular systems frequently suffer from "early stopping"—halting several meters before a target due to poor distance estimation. This research aims to provide these robots with a "spatial sense" using zero-shot metric depth estimation, avoiding the need for expensive LiDAR sensors.
Methodology
The MDE-AgriVLN framework augments the standard navigation pipeline with a Monocular Depth Estimation module.
- Monocular Depth Estimation: The system processes RGB images through Depth Pro, a zero-shot metric depth estimator. It generates a Depth Matrix (a 16x9 grid of absolute distances in meters) and a Depth Map (a color-coded visualization of the 3D structure).
- Multimodal Reasoning: These features are integrated into a Vision-Language Model (Qwen2.5-VL-32B) paired with a reasoning model (DeepSeek-R1-32B).
- Action Selection: The model generates a chain-of-thought reasoning process based on the instruction, RGB image, and depth data to select actions like
FORWARDorSTOPbased on calculated proximity to targets.
Evaluation and Results
The model was evaluated on the Agriculture-to-Agriculture (A2A) benchmark, covering 1,560 episodes across diverse scenes including farms, greenhouses, and forests.
Key Metrics
- Success Rate (SR): MDE-AgriVLN achieved a success rate of 0.32, a 39% relative improvement over the 0.23 baseline.
- Navigation Error (NE): The average stopping distance from the target was reduced from 4.43m to 4.08m.
Insights
- Metric Depth Importance: Absolute metric depth was found to be significantly more effective than relative depth for satisfying specific "stop" conditions in instructions.
- Environment Complexity: The impact of MDE was most pronounced in chaotic environments like forests, where visual size is a poor proxy for distance due to irregular tree sizes.
- Alignment Burden: Interestingly, providing both a depth matrix and a depth map did not always outperform the matrix alone, suggesting that redundant spatial data can sometimes distract the VLM.
Conclusion and Future Work
MDE-AgriVLN demonstrates that integrating monocular depth estimation significantly enhances the spatial awareness of agricultural robots. Future work will address current limitations, including spatial misalignment between RGB and depth features, computational latency (currently 0.2s–0.8s per step), and robustness to ambiguous human instructions.
Summary updated