PlantVLM-Edge: Quantized Vision-Language Models for On-Device Crop Health Diagnosis

Gaurab Wagle, Bishal Adhikari, Jiajia Li, Xiaolei Huang, Dong ChenarXiv 2026 · arXiv

TL;DR

PlantVLM-Edge is a new framework that brings conversational crop health diagnosis to resource-constrained edge devices by combining domain-specific fine-tuning with aggressive model quantization. By optimizing models like Qwen3-VL-2B, the system can provide expert-level plant disease and pest management advice entirely offline, delivering diagnostic responses in under a second on an NVIDIA Jetson AGX Orin.

What Problem Does This Paper Address?

Agricultural productivity is constantly threatened by pests and diseases, which account for up to 40% of annual crop yield losses globally. While deep learning has made automated diagnosis highly accurate, traditional computer vision models (like CNNs or YOLO) typically only provide a class label or a bounding box. They can't explain why a plant is sick or tell a grower what to do about it.

Vision-Language Models (VLMs) offer a solution by allowing interactive, natural-language support. However, these models are massive, requiring significant memory and compute power. This makes them difficult to deploy in remote farming environments where internet connectivity is unreliable and cloud costs are prohibitive. The authors address this by creating a pipeline to adapt, compress, and serve these models directly on edge hardware.

Methods

The PlantVLM-Edge framework consists of three main stages:

  1. Domain Adaptation: The researchers started with two open-weight VLMs: Qwen3-VL-2B and LLaVA-1.5-7B. They used Low-Rank Adaptation (LoRA) to fine-tune the language backbones on agricultural data while keeping the vision encoders frozen.
  2. Post-Training Quantization (PTQ): To shrink the models, they tested four quantization schemes: Round-to-Nearest (RTN), Activation-aware Weight Quantization (AWQ), and SmoothQuant at both 8-bit and 4-bit precision. This process reduces the numerical precision of the model weights to save memory and speed up inference.
  3. Edge Deployment: The quantized models were served using the vLLM engine on an NVIDIA Jetson AGX Orin. A browser-based chat interface (Open WebUI) allows users to interact with the model over a local network without needing an internet connection.
Figure 2: Overview of the PlantVLM-Edge framework. A pretrained vision-language model is adapted to plant-disease and insect-pest question answering with LoRA, with the vision encoder and projector kept frozen. The fine-tuned model is then quantized into four configurations and served with vLLM on an NVIDIA Jetson AGX Orin, where its weight footprint and latency are measured. Growers interact with the deployed model offline through a browser-based chat interface.

Datasets

The models were trained and evaluated on two primary agricultural VQA datasets:

  • PlantExpertVQA: A massive dataset for plant disease diagnosis (38 crops, 89 diseases). The authors used a stratified subset of ~63,500 question-answer pairs.
  • IP-VQA: A dataset for insect pest identification covering 93 pest categories and various management practices.
Figure 1: Representative image-question-answer pairs from the PlantExpertVQA and IP-VQA datasets.

Key Insights and Ideas

  • Adaptation is the Decisive Step: Domain fine-tuning provided a massive leap in quality (e.g., raising ROUGE-L scores from ~15 to ~78), whereas quantization resulted in only minor quality drops.
  • The "Small Model" Advantage: Interestingly, the fine-tuned 2B model performed just as well as the 7B model in aggregate answer quality. Because the 2B model is inherently faster and smaller, it is the clear winner for edge deployment.
  • Quantization Sweet Spot: 8-bit quantization was found to be effectively lossless. At 4-bit, AWQ significantly outperformed standard Round-to-Nearest (RTN) on the smaller 2B model, recovering much of the loss in long, explanatory answers.
  • Hardware Bottlenecks: The researchers noted that while weight quantization speeds up the "decoding" phase (generating text), the "prefill" phase (processing the image) is still limited by the vision encoder, which remained at 16-bit precision.

Key Results

  • Efficiency: The 4-bit Qwen3-VL-2B model required only 2.01 GiB of weight memory (a 49% reduction from the 16-bit version).
  • Latency: On the Jetson AGX Orin, the recommended configuration (W8A16-RTN) returns its first token in 0.12 seconds and completes a full 64-token diagnostic response in just 0.78 seconds.
  • Quality Retention: 4-bit AWQ retained 98.6% of the full-precision model's answer quality across both diagnostic domains.
  • Backbone Comparison: The 2B model was consistently 2.4–3.6x faster than the 7B model at equal precision, proving that architecture choice matters more for edge efficiency than the specific quantization method.

Limitations and Future Work

While PlantVLM-Edge is a significant step forward, the authors identify several areas for improvement:

  • Vision Encoder Compression: The vision encoder remains a bottleneck for initial response time (TTFT). Future work will explore quantizing the vision pathway.
  • Expert Validation: While NLP metrics (like ROUGE-L) are useful, the authors suggest that future studies should include expert human evaluation to ensure diagnostic correctness in real-world scenarios.
  • Hands-Free Interaction: The current system uses a web UI, but the authors envision integrating local speech-to-text to allow growers to use the system hands-free in the field.
  • Lower-Cost Hardware: Testing on even more constrained devices, such as the Jetson Orin Nano, is a planned next step.

Summary updated

Abstract

Timely and accurate crop health diagnosis is essential for sustainable agricultural production, yet accurate diagnosis often requires expert knowledge that is not readily available. Recent vision-language models (VLMs) have demonstrated remarkable capabilities in visual understanding, reasoning, and natural-language interaction, creating new opportunities for intelligent crop health assessment. However, deploying VLMs on resource-constrained devices remains challenging due to their substantial computational and memory requirements. This paper presents PlantVLM-Edge, a quantized vision-language framework for conversational crop health diagnosis that can run entirely on resource-constrained edge devices. PlantVLM-Edge combines agricultural domain adaptation, post-training quantization, and on-device serving with an offline chat interface, allowing growers to ask open-ended questions and receive diagnoses, symptom explanations, and management guidance without cloud connectivity. The framework is backbone-agnostic: we validate it with two different VLM types with varying scale and architecture (Qwen3-VL-2B and LLaVA-1.5-7B), four quantization schemes, and two diagnostic domains (plant diseases and insect pests), and we deploy all variants on an NVIDIA Jetson AGX Orin. Eight-bit quantization affects answer quality minimally relative to the full-precision model, and 4-bit activation-aware weight quantization (AWQ) retains 98.6% of full-precision answer quality on both domains while reducing the weight memory of the 2B backbone by 49%. In its most compact form, PlantVLM-Edge requires 2.01 GiB of weight memory, returns its first token in about 0.12 s, and completes a 64-token diagnostic response in 0.78 s. These results demonstrate that timely and accurate crop health diagnosis is practical on edge devices, providing a foundation for next-generation crop monitoring and decision support systems.
BibTeX
@misc{wagle2026plantvlm,
  author = {Gaurab Wagle and Bishal Adhikari and Jiajia Li and Xiaolei Huang and Dong Chen},
  title = {{PlantVLM-Edge: Quantized Vision-Language Models for On-Device Crop Health Diagnosis}},
  year = {2026},
  eprint = {2611.03557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03557},
}
Figure