Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild

Tianqi Wei, Zhi Chen, Xin Yu

Abstract

Plant disease recognition is a critical task that ensures crop health and mitigates the damage caused by diseases. A handy tool that enables farmers to receive a diagnosis based on query pictures or the text description of suspicious plants is in high demand for initiating treatment before potential diseases spread further. In this paper, we develop a multimodal plant disease image retrieval system to support disease search based on either image or text prompts. Specifically, we utilize the largest in-the-wild plant disease dataset PlantWild, which includes over 18,000 images across 89 categories, to provide a comprehensive view of potential diseases relating to the query. Furthermore, cross-modal retrieval is achieved in the developed system, facilitated by a novel CLIP-based vision-language model that encodes both disease descriptions and disease images into the same latent space. Built on top of the retriever, our retrieval system allows users to upload either plant disease images or disease descriptions to retrieve the corresponding images with similar characteristics from the disease dataset to suggest candidate diseases for end users' consideration.

Introduction

Plant diseases represent a significant global threat, resulting in over $200 billion in annual economic losses and undermining food security. While machine learning has advanced disease identification, existing models often fail in "in-the-wild" conditions due to complex backgrounds and varying lighting. Furthermore, most current tools are unimodal, supporting only image queries. This ignores scenarios where a farmer might observe symptoms—such as "wilting flowers"—that are easier to describe than to photograph clearly. To bridge these gaps, we present Snap’n Diagnose, a multimodal retrieval system designed for real-world agricultural environments.

Methodology and System Architecture

The Snap’n Diagnose system shifts from traditional classification to a retrieval-based approach. By showing users similar images from a verified database, the system provides actionable insights for visual confirmation.

Overview of the Snap'n Diagnose model architecture and application workflow.

Shared Latent Space

The core of the system is a vision-language model based on CLIP (Contrastive Language-Image Pre-training). We project both images and textual descriptions into a shared mathematical "latent space," allowing the system to calculate cosine similarity between a text query and an image database (or vice versa).

MVPDR Integration

To handle the complexities of outdoor environments, we incorporate the Multimodal Plant Disease Recognition (MVPDR) method. This includes a fine-tuned Multi-Layer Perceptron (MLP) that adapts the model to the specific nuances of plant disease features. The system is deployed via a Flask-based backend, where visual and text features are pre-extracted to ensure low-latency search results.

The PlantWild Dataset

The system is powered by PlantWild, the world’s largest "in-the-wild" plant disease dataset. It contains over 18,000 images across 89 disease categories. Unlike laboratory datasets, PlantWild captures the messy reality of farm conditions. Crucially, it includes diverse textual descriptions for each disease, enabling the cross-modal search capabilities that define our system.

Experimental Results and Interface

We evaluated Snap’n Diagnose against a standard Zero-shot CLIP baseline. Our system demonstrated substantial improvements across all key metrics, particularly in Top-1 accuracy, which saw a nearly 27% increase.

Metric Zero-shot CLIP Snap’n Diagnose (Ours)
Top-1 Accuracy 40.92% 67.32%
Top-5 Accuracy 65.75% 80.65%
Top-10 Accuracy 74.81% 88.11%
mAP (mean Average Precision) 68.72% 79.34%
The Snap'n Diagnose web interface showing a query image and ranked retrieval results with similarity scores.

The web interface allows users to upload images or type symptom descriptions. As shown in the results, the system successfully retrieves relevant disease matches even for complex visual symptoms, providing users with ranked candidates and similarity scores.

Conclusion

Snap’n Diagnose offers a robust, multimodal solution for real-world plant disease identification. By combining the scale of the PlantWild dataset with a specialized CLIP-based architecture, we provide farmers with a flexible tool for early disease detection. Future work will focus on expanding the disease database and developing offline capabilities for remote areas with limited connectivity.

Summary updated

Abstract

Plant disease recognition is a critical task that ensures crop health and mitigates the damage caused by diseases. A handy tool that enables farmers to receive a diagnosis based on query pictures or the text description of suspicious plants is in high demand for initiating treatment before potential diseases spread further. In this paper, we develop a multimodal plant disease image retrieval system to support disease search based on either image or text prompts. Specifically, we utilize the largest in-the-wild plant disease dataset PlantWild, which includes over 18,000 images across 89 categories, to provide a comprehensive view of potential diseases relating to the query. Furthermore, cross-modal retrieval is achieved in the developed system, facilitated by a novel CLIP-based vision-language model that encodes both disease descriptions and disease images into the same latent space. Built on top of the retriever, our retrieval system allows users to upload either plant disease images or disease descriptions to retrieve the corresponding images with similar characteristics from the disease dataset to suggest candidate diseases for end users' consideration.
BibTeX
@inproceedings{wei2024snap,
  author = {Tianqi Wei and Zhi Chen and Xin Yu},
  title = {{Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild}},
  booktitle = {MMAsia},
  year = {2024},
  doi = {10.1145/3696409.3700293},
}
Figure