AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding

Haocheng Li, Juepeng Zheng, Zenghao Yang, Kaiqi Du, Guilong Xiao, Gengmeng Pu, Haohuan Fu, Jianxi HuangarXiv 2026 · arXiv

TL;DR

AgroVG is a new large-scale benchmark designed to evaluate how well AI systems can locate specific agricultural targets—like weeds, pests, or diseased leaves—using natural language instructions. By framing visual grounding as a "generalized set prediction" problem, it challenges models to find one, many, or zero targets in complex, dense field environments. Evaluation of 26 state-of-the-art models reveals that current AI severely struggles with target completeness and knowing when to abstain from pointing out non-existent features.

What Problem Does This Paper Address?

In precision agriculture, AI systems need to translate human-like instructions into physical actions—such as telling a robot to "remove all mature weeds" or "localize every yellowing leaf." This capability is known as visual grounding.

While general-domain AI models have made massive strides in locating everyday objects, agricultural environments present unique, messy challenges. Targets are often tiny, highly repetitive (e.g., a field of visually identical wheat heads), heavily occluded by foliage, or irregularly shaped. Furthermore, an instruction might refer to a single object, a whole cluster, or something completely absent from the image.

Existing benchmarks typically assume a single target per image and fail to test whether a model can completely find a whole set of objects or correctly do nothing when a target isn't there. AgroVG addresses this gap by introducing a rigorous evaluation framework tailored to the realities of close-range field and orchard imagery.

Figure 1: AgroVG at a glance. AgroVG casts agricultural grounding as generalized set prediction: a query over an agricultural image may require one target, multiple targets, or no output. The benchmark spans six target families and two output protocols, bounding boxes (T1) and instance masks (T2), to diagnose multi-target completeness, target-absent abstention, and mask grounding. Preprint.

Methods

The authors formulate agricultural visual grounding as a generalized set prediction problem. Given an image and a natural language query, a model must output a set of matching instances, which can contain one, multiple, or zero elements (requiring abstention).

AgroVG establishes two distinct tasks:

  1. T1: Bounding-Box Grounding: Models predict axis-aligned boxes for coarse localization.
  2. T2: Instance-Mask Grounding: Models predict pixel-precise binary masks, which is crucial for irregularly shaped targets like disease lesions or overlapping weeds.

To build the benchmark, the authors developed a split-aware construction pipeline:

  • Annotation Normalization: Heterogeneous data formats from various public datasets were unified into a single instance-level schema.
  • Expert Quality Control: Ten domain experts manually screened candidate images to eliminate misalignments, ambiguous targets, or corrupted images.
  • Template-Based Query Generation: Queries were programmatically generated based on verified ground-truth attributes (e.g., spatial relations, geometric ranks like "the largest tree", or set-level quantifications).

For evaluation, the benchmark uses a maximum-cardinality bipartite matching strategy for bounding boxes to ensure models are penalized for both missing targets and predicting duplicates. For masks, it measures query-level coverage rather than individual instance matching to accommodate diverse model architectures.

Datasets

AgroVG aggregates 10 public source datasets into a single benchmark containing 10,071 annotation-grounded image-query pairs. It spans six core agricultural target families:

  • Crop/Weed (PhenoBench, CropAndWeed)
  • Fruit (MinneApple, ACFR, MegaFruits)
  • Wheat Head (GWHD)
  • Pest (IP102)
  • Plant Disease (PlantSeg)
  • Tree Canopy (OAM-TCD, Dumortier et al.)

While Task 1 (boxes) covers all six families, Task 2 (masks) is currently restricted to crop/weed, fruit, and plant disease due to the availability of reliable pixel-level annotations in the source data.

Figure 4: Dataset statistics of AgroVG. Panels summarize source/family composition, query-regime counts, instance-density buckets, and target-size distributions for T1 and T2.

Key Insights and Ideas

  • Localization $\neq$ Set Completeness: A model can be excellent at finding a target, but terrible at finding all targets. In dense agricultural scenes, models frequently stop after identifying the most salient object, missing smaller or occluded instances.
  • The Hallucination Trap: Models are poorly calibrated for target-absent queries. When a query mentions a familiar category (e.g., "apples"), many models feel compelled to predict something even if there isn't an apple in the picture, driven by language bias rather than visual evidence.
  • The Mask Quality vs. Abstention Trade-off: There is a stark tension between a model's ability to generate high-quality masks and its ability to abstain. Models like SAM 3 excel at abstaining on empty queries but produce poor masks on positive queries, whereas specialized segmentation MLLMs (like LISA) generate great positive masks but hallucinate heavily on target-absent images.

Key Results

The authors conducted a zero-shot evaluation of 26 model configurations, including closed-source Multimodal Large Language Models (MLLMs), open-source Vision-Language Models (VLMs), and specialized grounding systems.

  • Open-Source VLMs Lead Box Grounding: Open-source models like InternVL3.5-38B and Qwen3-VL outperformed commercial heavyweights (like GPT-4o and Gemini 2.5 Pro) on bounding-box localization.
  • Low Multi-Target Completeness: The best multi-target Set-$F_1$ score at an Intersection-over-Union (IoU) threshold of 0.50 reached only 0.3540 (achieved by InternVL3.5-8B), showing that models miss a massive fraction of valid targets in dense scenes.
  • Poor Mask Precision: For pixel-level mask grounding (T2), the best positive-query mask success rate at a strict [email protected] threshold remained below 0.17 (PSALM), underscoring the difficulty of fine-grained agricultural segmentation.
Performance comparison of closed-source MLLMs, open-source VLMs, and specialized systems on Task 1.

Limitations and Future Work

The authors note several limitations in the current version of AgroVG:

  1. Limited Mask Coverage: Task 2 (masks) only covers three of the six target families because datasets for pests, wheat heads, and tree canopies lack audited, instance-level pixel annotations.
  2. Linguistic Diversity: Because queries are template-based to ensure exact target-set traceability, they lack the natural linguistic variety of human speech.
  3. Single-Reviewer Verification: Each image panel was reviewed by only one expert during quality control, though strict geometric criteria were enforced to mitigate errors.

Future work will focus on expanding query diversity, incorporating more mask-capable data sources, and exploring controlled fine-tuning strategies to bridge the massive performance gaps highlighted by the benchmark.

Summary updated

Abstract

Visual grounding, the task of localizing objects described by natural-language expressions, is a foundational capability for agricultural AI systems, enabling applications such as selective weeding, disease monitoring, and targeted harvesting. Reliable evaluation of agricultural visual grounding remains challenging because agricultural targets are often small, repetitive, occluded, or irregularly shaped, and instructions may refer to one, many, or no objects in an image. Evaluating this capability therefore requires jointly testing localization accuracy, target-set completeness, and existence-aware abstention. To address these challenges, we introduce \textbf{AgroVG}, a multi-source benchmark that formulates agricultural grounding as generalized set prediction: given an image and a referring expression, a model must return all matching target instances or abstain when no target is present. AgroVG contains 10{,}071 annotation-grounded image-query pairs from ten source datasets across six target families: crop/weed, fruit, wheat head, pest, plant disease, and tree canopy. It supports bounding-box grounding (T1) across all six families and instance-mask grounding (T2) on sources with reliable instance-level pixel annotations, with queries covering single-target, multi-target, and target-absent regimes. AgroVG further provides task-specific protocols for box-set matching and query-level mask coverage. Zero-shot evaluation of 26 model configurations spanning closed-source MLLMs, open-source VLMs, and specialized grounding systems reveals persistent gaps: the best multi-target Set-$F_1$ reaches only 0.35, and the best positive-query mask success rate at [email protected] remains below 0.17. Data and code are available at https://anonymous.4open.science/r/AgroVG-5172/ .
BibTeX
@misc{li2026agrovg,
  author = {Haocheng Li and Juepeng Zheng and Zenghao Yang and Kaiqi Du and Guilong Xiao and Gengmeng Pu and Haohuan Fu and Jianxi Huang},
  title = {{AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding}},
  year = {2026},
  eprint = {2605.22034},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.22034},
}

Loading related content...

Figure