Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild

Tianqi Wei, Zhi Chen, Xin Yu

TL;DR

Snap and Diagnose is a multimodal retrieval system designed to identify plant diseases in real-world, "in-the-wild" conditions. By leveraging a shared latent space for both images and text descriptions, the system allows users to search for potential diseases using either a photo of an affected plant or a simple text description of symptoms (e.g., "yellow spots on leaves"), significantly outperforming standard vision-language models.

What Problem Does This Paper Address?

Plant diseases are a massive global threat, causing over $200 billion in annual economic losses and threatening food security. While machine learning has shown promise in identifying these diseases, two major hurdles remain:

  1. The "In-the-Wild" Gap: Most existing models are trained on "in-laboratory" datasets with clean backgrounds and perfect lighting. Their performance plummets when faced with the messy reality of a farm—varying weather, complex backgrounds, and different camera angles.
  2. Unimodal Limitations: Most current tools only support image queries. However, a farmer might observe a symptom (like "wilting flowers") that is difficult to photograph clearly but easy to describe. There is a lack of systems that can handle both images and text interchangeably to find matches in a disease database.

Methods

The researchers developed a system called Snap’n Diagnose, which utilizes a retrieval-based approach rather than simple classification.

  • Shared Latent Space: The core of the system is a vision-language model based on CLIP (Contrastive Language-Image Pre-training). By projecting both images and textual descriptions into the same mathematical "latent space," the system can calculate the similarity between a text query and a database of images (or vice versa).
  • MVPDR Integration: To handle the "in-the-wild" aspect, the authors incorporate the MVPDR (Multimodal Plant Disease Recognition) method. This includes a fine-tuned Multi-Layer Perceptron (MLP) that helps the model adapt specifically to the nuances of plant disease features across different environments.
  • System Architecture: The system is built with a Flask-based backend and a web-based frontend. To ensure a fast user experience, visual and text features are pre-extracted and stored, reducing the computational load during a live search. When a user submits a query, the system performs a cosine similarity search and returns a ranked list of the most relevant disease images.

Datasets

The system relies on PlantWild, which the authors describe as the world’s largest "in-the-wild" plant disease dataset.

  • Scale: Over 18,000 images.
  • Diversity: 89 different disease categories.
  • Multimodality: Crucially, PlantWild provides diverse textual descriptions for each disease type, which is what enables the cross-modal (text-to-image) search capability.

Key Insights and Ideas

  • Retrieval vs. Classification: Instead of the model simply saying "This is Disease X," the retrieval approach shows the user the most similar images from a verified database. This provides "actionable insights," allowing the user to visually confirm the diagnosis against known cases.
  • The Power of Fine-Tuning: While "zero-shot" models like standard CLIP are powerful, they struggle with specialized domains like botany. By adding the MVPDR fine-tuned layers, the authors significantly improved the model's ability to distinguish between subtle disease symptoms in complex outdoor settings.
  • Accessibility: By supporting text queries, the system lowers the barrier to entry for farmers who may not have high-quality cameras or who are dealing with symptoms that are hard to capture in a single frame.

Key Results

The authors compared Snap’n Diagnose against a baseline of Zero-shot CLIP [8]. The improvements were substantial across all metrics:

Metric Zero-shot CLIP Snap’n Diagnose (Ours)
Top-1 Accuracy 40.92% 67.32%
Top-5 Accuracy 65.75% 80.65%
Top-10 Accuracy 74.81% 88.11%
mAP (mean Average Precision) 68.72% 79.34%

The nearly 27% jump in Top-1 accuracy demonstrates that the specialized MLP and the use of the PlantWild dataset are highly effective for real-world applications.

Limitations and Future Work

While the paper presents a robust system, a few areas remain for further exploration:

  • Diagnostic vs. Suggestive: The system is a retrieval tool meant to "suggest candidate diseases for end users’ consideration." It still requires a human in the loop to make the final call, which might be a limitation for fully automated agricultural systems.
  • Dataset Expansion: While 89 classes is significant, there are thousands of plant diseases globally. Expanding the database to include more rare or regional diseases would be a logical next step.
  • Offline Capability: The current implementation uses a Flask API (requiring an internet connection). For remote farmers with poor connectivity, an edge-computing or offline version of the model would be highly beneficial.

Summary updated

Abstract

Plant disease recognition is a critical task that ensures crop health and mitigates the damage caused by diseases. A handy tool that enables farmers to receive a diagnosis based on query pictures or the text description of suspicious plants is in high demand for initiating treatment before potential diseases spread further. In this paper, we develop a multimodal plant disease image retrieval system to support disease search based on either image or text prompts. Specifically, we utilize the largest in-the-wild plant disease dataset PlantWild, which includes over 18,000 images across 89 categories, to provide a comprehensive view of potential diseases relating to the query. Furthermore, cross-modal retrieval is achieved in the developed system, facilitated by a novel CLIP-based vision-language model that encodes both disease descriptions and disease images into the same latent space. Built on top of the retriever, our retrieval system allows users to upload either plant disease images or disease descriptions to retrieve the corresponding images with similar characteristics from the disease dataset to suggest candidate diseases for end users' consideration.
BibTeX
@inproceedings{wei2024snap,
  author = {Tianqi Wei and Zhi Chen and Xin Yu},
  title = {{Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild}},
  booktitle = {MMAsia},
  year = {2024},
  doi = {10.1145/3696409.3700293},
}

Loading related content...

Figure