AgriGPT-VL: Agricultural Vision–Language Understanding Suite

Bo Yang, Yunkui Chen, Lanfei Feng, Yu Zhang, Xiao Xu, Jianyu Zhang, Nueraili Aierken, Runhe Huang, Hongjian Lin, Yibin Ying, Shijian LiSubmission to ICLR 2026 · Withdrawn

TL;DR

AgriGPT-VL is an open-source multimodal framework tailored for agriculture, featuring Agri-3M-VL (a 3-million-instance multimodal dataset), a progressive curriculum training pipeline, and AgriBench-VL-4K (a challenging multimodal benchmark). Built on a 7B-parameter base, AgriGPT-VL outperforms general-purpose flagship models like Qwen2.5-VL-72B and Gemini 2.5 Pro on agricultural visual reasoning without degrading its general or text-only capabilities.

What Problem Does This Paper Address?

While multimodal large language models (MLLMs) excel on web-scale images of common scenes, they struggle in specialized scientific domains like agriculture. Real-world agronomic tasks—such as fine-grained pest identification, foliar disease diagnosis, weed recognition, and treatment planning—require nuanced domain expertise absent from standard web crawls.

Existing agricultural AI has historically suffered from fragmentation: models were either strictly text-based (e.g., AgriBERT, AgriGPT) or limited to narrow classification datasets (e.g., PlantVillage). Meanwhile, general frontier vision-language models frequently hallucinate or confuse visually similar pests and crop symptoms. AgriGPT-VL resolves this by providing an end-to-end open ecosystem: scalable data generation, curriculum-driven model training, and rigorous evaluation.

Methods

The authors adopt a multi-agent data generation framework alongside a progressive curriculum training schedule:

[Raw Agricultural Images] (1.06M images / 42k classes)
           │
           ▼
[Multi-Agent Data Generator] 
   1. Structured Captions (Growth stage, lesion color, morphology)
   2. Instruction Synthesis (Qwen2.5-VL-72B / GPT-4o)
   3. Multi-Agent Refinement (Feedback, Evaluation, Rethinking loop)
   4. Relevance & Quality Filtering
           │
           ▼
[Agri-3M-VL Dataset] ──► Progressive Curriculum:
                         • Stage 1: Text Grounding (Continual PT + Agri-342K SFT)
                         • Stage 2a: Shallow Multimodal (Frozen vision & LLM, train adapter)
                         • Stage 2b: Deep Multimodal (LoRA on vision stack & LLM on 2M VQA)
                         • Stage 2c: GRPO (Reinforcement learning with exact & semantic rewards)
  1. Multi-Agent Data Generation: Raw agricultural images are transformed into instruction data through four steps: structured attribute captioning, diverse instruction synthesis, a multi-agent verification loop (Feedback, Evaluation, and Rethinking agents), and image-grounding filtering to remove visual hallucinations.
  2. Progressive Curriculum Training:
  3. Stage 1 (Text-only Grounding): Initializes from Qwen2.5-VL, performing continual pretraining on ~2.2B agricultural tokens followed by instruction tuning on Agri-342K.
  4. Stage 2a (Shallow Alignment): Keeps the vision encoder and LLM frozen while training only the connector on 1M image–caption pairs to establish cross-modal semantic anchors.
  5. Stage 2b (Deep Alignment): Unfreezes the vision encoder and LLM backbone via LoRA on 2M image–QA instances covering diagnostic and multi-hop reasoning.
  6. Stage 2c (GRPO Refinement): Applies Group Relative Policy Optimization on 15k reinforcement learning instances using a hybrid reward: a binary exact-match reward and a continuous semantic similarity reward (combining BLEU, METEOR, and ROUGE-L).

Datasets

  • Raw Image Aggregation: Over 1.06 million images spanning 42,253 fine-grained agricultural classes gathered from PlantVillage, Species196, Insect Foundation, SelectDataset, and fruit quality datasets.
  • Agri-3M-VL (Training): The largest open agricultural multimodal corpus to date, comprising 1M image–caption pairs, 2M image-grounded VQA pairs, 50K expert-level instruction pairs, and 15K preference-optimization pairs for GRPO.
  • AgriBench-VL-4K (Evaluation): A new evaluation suite consisting of 2,018 open-ended VQA pairs and 1,858 single-choice questions. It introduces an $\text{Acc}^+$ metric where two complementary questions are evaluated per image; both must be answered correctly to credit the model, heavily penalizing random guessing.

Key Insights and Ideas

  • Curricular Isolation Prevents Catastrophic Forgetting: Consolidating domain knowledge via text-only grounding prior to unfreezing visual representations prevents the model from degrading on text-based agricultural reasoning (tested on AgriBench-13K) or general benchmarks like MMLU.
  • Reinforcement Learning Tightens Visual Consistency: In ablation studies, adding GRPO delivered the largest boost on the strict $\text{Acc}^+$ metric (jumping from 66.67% to 74.17%), demonstrating that preference optimization grounds model outputs firmly in image evidence rather than superficial statistical correlations.
  • Cross-Domain Generalization: Domain-specific tuning did not impair broader capabilities; in fact, visual reasoning generalized outward, lifting base performance on MMMU from 0.4329 to 0.4599.

Key Results

  • AgriBench-VL-4K: AgriGPT-VL achieved 85.84% accuracy (Acc) and 74.17% strict cross-consistency ($\text{Acc}^+$), outperforming 70B+ parameter baselines including Qwen2.5-VL-72B-Instruct (81.70% Acc / 67.49% $\text{Acc}^+$) and proprietary models like Gemini 2.5 Pro (81.59% Acc / 66.74% $\text{Acc}^+$).
  • Text Generation Quality: On open-ended responses, AgriGPT-VL achieved 26.27 BLEU and 47.55 METEOR, substantially ahead of Qwen2.5-VL-72B (15.41 BLEU / 41.38 METEOR).
  • Pairwise Preference: In blind head-to-head LLM-as-a-judge evaluations, AgriGPT-VL secured win rates of 62% against Qwen2.5-VL-72B, 89% against InternVL3-14B, and 99% against LLaVA-1.5-7B.

Limitations and Future Work

  • Simulated vs. Real-World Field Conditions: The training and evaluation benchmarks are based on static, curated image sets. The paper does not evaluate performance under real-world farming deployment challenges such as extreme occlusions, varying lighting, low-bandwidth drone feeds, or video streams.
  • Base Model Scale: Experiments center on adapting a 7B-parameter base; scaling the curriculum framework to larger architectures or distilling down to ultra-compact edge models (<3B) for on-device farm machinery remains unaddressed.

Summary updated

Abstract

Despite rapid advances in multimodal large language models, agricultural applications remain constrained by the scarcity of domain-tailored models, curated vision–language corpora, and rigorous evaluation. To address these challenges, we present the AgriGPT-VL Suite, a unified multimodal framework for agriculture. Our contributions are threefold. First, we introduce Agri-3M-VL, the largest vision–language corpus for agriculture to our knowledge, curated by a scalable multi-agent data generator; it comprises 1M image–caption pairs, 2M image-grounded VQA pairs, 50K expert-level VQA instances, and 15k GRPO reinforcement learning dataset. Second, we develop AgriGPT-VL, an agriculture-specialized vision–language model trained via a progressive curriculum of textual grounding, multimodal shallow/deep alignment, and GRPO refinement. This method achieves strong multimodal reasoning while preserving text-only capability. Third, we establish AgriBench-VL-4K, a compact yet challenging evaluation suite with open-ended and image-grounded questions, paired with a multi-metric evaluation and an LLM-as-a-judge framework. Experiments show that AgriGPT-VL outperforms leading general-purpose VLMs on AgriBench-VL-4K, achieving higher pairwise win rates in the LLM-as-a-judge evaluation. Meanwhile, it remains competitive on the text-only AgriBench-13K with no noticeable degradation of language ability. Ablation studies further confirm consistent gains from our alignment and GRPO refinement stages. All resources will be released to support reproducible research and deployment in low-resource agricultural settings at: https://anonymous.4open.science/r/AgriGPT-VL-DA65/
BibTeX
@misc{yang2025agrigpt,
  author = {Bo Yang and Yunkui Chen and Lanfei Feng and Yu Zhang and Xiao Xu and Jianyu Zhang and Nueraili Aierken and Runhe Huang and Hongjian Lin and Yibin Ying and Shijian Li},
  title = {{AgriGPT-VL: Agricultural Vision–Language Understanding Suite}},
  year = {2025},
  eprint = {2510.04002},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.04002},
}
Figure