Agri-MRAG: Parameter-Enhanced Multimodal Retrieval-Augmented Generation for Agricultural Visual Question Answering

Fuyong Liu, Xiantao Lei, Xining ZhangarXiv 2026 · arXiv

TL;DR

Agri-MRAG is a novel framework that significantly boosts the accuracy of Agricultural Visual Question Answering (Agri-VQA) by moving beyond simple text-based retrieval. It introduces an adaptive clustering mechanism to filter noise, injects retrieved knowledge directly into model parameters via fused LoRA adapters, and employs an uncertainty-based routing system to balance inference speed with diagnostic reliability.

What Problem Does This Paper Address?

Multimodal Large Language Models (MLLMs) are powerful, but they often "hallucinate" when faced with specialized domains like agriculture. Identifying a specific crop disease from a leaf image requires niche pathological knowledge that general-purpose models lack.

While Multimodal Retrieval-Augmented Generation (MRAG) helps by fetching external facts, current methods suffer from two major flaws:

  1. Noisy Retrieval: They often use a fixed number of retrieved items ($k$), which might include redundant or irrelevant information that distracts the model.
  2. Weak Integration: Most RAG systems simply concatenate retrieved text into the prompt. This "prompt-level" fusion is shallow and can lead to "attention distraction" if the context becomes too long or noisy.

Methods

The authors propose Agri-MRAG, which features three innovative components:

  • Semantic-Consistency-Guided Clustering: Instead of just grabbing the top $k$ results, the system groups retrieved image-text pairs into clusters in a multimodal embedding space. It then selects the most relevant cluster, allowing the retrieval size $k$ to adapt to the complexity of the question while filtering out high-confidence outliers.
  • Parameterized Knowledge Injection: This is the "secret sauce." Rather than just reading the retrieved text, the model uses pre-trained LoRA (Low-Rank Adaptation) adapters for each knowledge sample. During inference, the adapters corresponding to the retrieved evidence are dynamically fused and injected into the model’s Feed-Forward Network (FFN) layers. This allows the external knowledge to influence the model at a parameter level.
  • Perplexity-Guided (PPL) Routing: To save on computation, the model first attempts to answer using its internal parameters. If the perplexity (a measure of uncertainty) is low, it provides the answer immediately. If the model is "confused" (high PPL), it triggers a second, more intensive pass that incorporates explicit retrieved text into the prompt for deeper reasoning.

Datasets

The researchers evaluated their framework on two benchmarks:

  1. PlantVillageVQA: A large-scale, public dataset containing over 193,000 QA pairs across 14 crop species, primarily focused on leaf diseases.
  2. PlantVQA (New): A region-specific dataset constructed for this study. it contains 7,514 images and 15,028 expert-verified QA pairs focusing on forestry and fruit crops (jujube, pear, apple, citrus, grape) in the Xinjiang region of China. It includes complex scenarios like pest damage traces and fruit-level symptoms.

Key Insights and Ideas

  • Parameters > Prompts: The paper demonstrates that "internalizing" knowledge via adapters is more effective than just "showing" the model text in a prompt. This deep coupling helps the model handle specialized terminology and fine-grained visual features more robustly.
  • Adaptive Retrieval Matters: Fixed-size retrieval is a bottleneck. By using clustering to determine the optimal volume of evidence, the model significantly improves its signal-to-noise ratio.
  • Efficiency through Uncertainty: The PPL-routing mechanism shows that we don't always need the "heavy machinery" of RAG. Many queries can be handled by the parameter-injected model alone, reserving explicit context expansion for truly difficult cases.

Key Results

Agri-MRAG outperformed several state-of-the-art baselines, including general MLLMs (Qwen3-VL) and agriculture-specific models (AgriChat).

  • ROUGE-L Scores: Achieved 41.21 on PlantVillageVQA and 46.58 on PlantVQA.
  • Vs. Baselines: It outperformed the standard mRAG by 11.70 and 4.51 points, and VisRAG by 8.80 and 3.39 points on the respective datasets.
  • Ablation Success: The clustering module was identified as the primary driver of the performance leap, fundamentally suppressing noise that usually plagues retrieval systems.

Limitations and Future Work

Despite the performance gains, the authors acknowledge a few hurdles:

  • Storage Overhead: Storing a LoRA adapter for every single knowledge sample is resource-intensive. For PlantVillageVQA, the adapter database reached 801 GB. Future work will look into pruning these to only keep adapters for high-frequency or critical samples.
  • Visual Alignment: The model still struggles with some fine-grained visual tasks, such as pinpointing the exact location of rot on a fruit.
  • Scalability: The current system is a proof-of-concept for specific regions; expanding the agricultural knowledge base to cover more global crop species and growth stages is the next step.

Summary updated

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) has shown promise for Agricultural Visual Question Answering (Agri-VQA), but existing methods still face challenges in retrieving relevant knowledge while avoiding redundancy and in integrating retrieved evidence beyond simple prompt concatenation. To address these issues, we propose Agri-MRAG, a multimodal retrieval-augmented framework for agricultural VQA. Specifically, we first introduce a semantic-consistency-guided clustering mechanism that combines multimodal clustering and weighted relevance scoring to select high-confidence knowledge subsets and adaptively determine the retrieval size $k$. We then design a parameterized knowledge injection strategy, where retrieved agricultural knowledge is encoded into LoRA adapters and merged into the feed-forward network (FFN) layers of the MLLM, allowing external knowledge to influence generation through parameter-level adaptation. In addition, a perplexity-guided (PPL-guided) routing mechanism is employed to switch between implicit parametric reasoning and explicit retrieval-assisted generation according to model uncertainty, balancing inference efficiency and answer reliability. For evaluation, we use PlantVillageVQA as a broad agricultural benchmark and construct PlantVQA, a region-specific forestry and fruit crop VQA dataset containing 7,514 images and 15,028 expert-verified question-answer pairs. Experiments show that Agri-MRAG achieves ROUGE-L scores of 41.21 and 46.58 on the two datasets, outperforming mRAG by 11.70 and 4.51 points, and VisRAG by 8.80 and 3.39 points, respectively.
BibTeX
@misc{liu2026agri,
  author = {Fuyong Liu and Xiantao Lei and Xining Zhang},
  title = {{Agri-MRAG: Parameter-Enhanced Multimodal Retrieval-Augmented Generation for Agricultural Visual Question Answering}},
  year = {2026},
  eprint = {2611.18771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18771},
}
Figure