TL;DR
AgroVG is a new large-scale benchmark designed to evaluate how well AI systems can locate specific agricultural targets—like weeds, pests, or diseased leaves—using natural language instructions. By framing visual grounding as a "generalized set prediction" problem, it challenges models to find one, many, or zero targets in complex, dense field environments. Evaluation of 26 state-of-the-art models reveals that current AI severely struggles with target completeness and knowing when to abstain from pointing out non-existent features.
What Problem Does This Paper Address?
In precision agriculture, AI systems need to translate human-like instructions into physical actions—such as telling a robot to "remove all mature weeds" or "localize every yellowing leaf." This capability is known as visual grounding.
While general-domain AI models have made massive strides in locating everyday objects, agricultural environments present unique, messy challenges. Targets are often tiny, highly repetitive (e.g., a field of visually identical wheat heads), heavily occluded by foliage, or irregularly shaped. Furthermore, an instruction might refer to a single object, a whole cluster, or something completely absent from the image.
Existing benchmarks typically assume a single target per image and fail to test whether a model can completely find a whole set of objects or correctly do nothing when a target isn't there. AgroVG addresses this gap by introducing a rigorous evaluation framework tailored to the realities of close-range field and orchard imagery.
Methods
The authors formulate agricultural visual grounding as a generalized set prediction problem. Given an image and a natural language query, a model must output a set of matching instances, which can contain one, multiple, or zero elements (requiring abstention).
AgroVG establishes two distinct tasks:
- T1: Bounding-Box Grounding: Models predict axis-aligned boxes for coarse localization.
- T2: Instance-Mask Grounding: Models predict pixel-precise binary masks, which is crucial for irregularly shaped targets like disease lesions or overlapping weeds.
To build the benchmark, the authors developed a split-aware construction pipeline:
- Annotation Normalization: Heterogeneous data formats from various public datasets were unified into a single instance-level schema.
- Expert Quality Control: Ten domain experts manually screened candidate images to eliminate misalignments, ambiguous targets, or corrupted images.
- Template-Based Query Generation: Queries were programmatically generated based on verified ground-truth attributes (e.g., spatial relations, geometric ranks like "the largest tree", or set-level quantifications).
For evaluation, the benchmark uses a maximum-cardinality bipartite matching strategy for bounding boxes to ensure models are penalized for both missing targets and predicting duplicates. For masks, it measures query-level coverage rather than individual instance matching to accommodate diverse model architectures.
Datasets
AgroVG aggregates 10 public source datasets into a single benchmark containing 10,071 annotation-grounded image-query pairs. It spans six core agricultural target families:
- Crop/Weed (PhenoBench, CropAndWeed)
- Fruit (MinneApple, ACFR, MegaFruits)
- Wheat Head (GWHD)
- Pest (IP102)
- Plant Disease (PlantSeg)
- Tree Canopy (OAM-TCD, Dumortier et al.)
While Task 1 (boxes) covers all six families, Task 2 (masks) is currently restricted to crop/weed, fruit, and plant disease due to the availability of reliable pixel-level annotations in the source data.
Key Insights and Ideas
- Localization $\neq$ Set Completeness: A model can be excellent at finding a target, but terrible at finding all targets. In dense agricultural scenes, models frequently stop after identifying the most salient object, missing smaller or occluded instances.
- The Hallucination Trap: Models are poorly calibrated for target-absent queries. When a query mentions a familiar category (e.g., "apples"), many models feel compelled to predict something even if there isn't an apple in the picture, driven by language bias rather than visual evidence.
- The Mask Quality vs. Abstention Trade-off: There is a stark tension between a model's ability to generate high-quality masks and its ability to abstain. Models like SAM 3 excel at abstaining on empty queries but produce poor masks on positive queries, whereas specialized segmentation MLLMs (like LISA) generate great positive masks but hallucinate heavily on target-absent images.
Key Results
The authors conducted a zero-shot evaluation of 26 model configurations, including closed-source Multimodal Large Language Models (MLLMs), open-source Vision-Language Models (VLMs), and specialized grounding systems.
- Open-Source VLMs Lead Box Grounding: Open-source models like InternVL3.5-38B and Qwen3-VL outperformed commercial heavyweights (like GPT-4o and Gemini 2.5 Pro) on bounding-box localization.
- Low Multi-Target Completeness: The best multi-target Set-$F_1$ score at an Intersection-over-Union (IoU) threshold of 0.50 reached only 0.3540 (achieved by InternVL3.5-8B), showing that models miss a massive fraction of valid targets in dense scenes.
- Poor Mask Precision: For pixel-level mask grounding (T2), the best positive-query mask success rate at a strict [email protected] threshold remained below 0.17 (PSALM), underscoring the difficulty of fine-grained agricultural segmentation.
Limitations and Future Work
The authors note several limitations in the current version of AgroVG:
- Limited Mask Coverage: Task 2 (masks) only covers three of the six target families because datasets for pests, wheat heads, and tree canopies lack audited, instance-level pixel annotations.
- Linguistic Diversity: Because queries are template-based to ensure exact target-set traceability, they lack the natural linguistic variety of human speech.
- Single-Reviewer Verification: Each image panel was reviewed by only one expert during quality control, though strict geometric criteria were enforced to mitigate errors.
Future work will focus on expanding query diversity, incorporating more mask-capable data sources, and exploring controlled fine-tuning strategies to bridge the massive performance gaps highlighted by the benchmark.
Summary updated