Integrating deep learning for visual question answering in Agricultural Disease Diagnostics: Case Study of Wheat Rust

Akash Nanavaty, Rishikesh Sharma, Bhuman Pandita, Omesh Goyal, Srinivas Rallapalli, Murari Mandal, Vaibhav Kumar Singh, Pratik Narang, Vinay ChamolaScientific 2024 · DOI

TL;DR

This paper introduces a novel approach to diagnosing wheat rust by combining deep learning with Visual Question Answering (VQA). They developed new datasets and a federated learning system that allows for accurate, context-aware disease identification, making advanced diagnostics more accessible to farmers.

What Problem Does This Paper Address?

Wheat rust is a major threat to global food security, causing significant crop losses worldwide. Traditional methods for diagnosing wheat rust rely on expert knowledge and are time-consuming, making rapid and widespread detection challenging. While deep learning has shown promise in image-based classification of plant diseases, there's a need for systems that can provide more nuanced, context-aware diagnostic information directly from images, akin to how an expert would interpret them. This paper aims to bridge this gap by developing an intelligent system that can answer specific questions about wheat rust from images, reducing reliance on human experts and enabling faster, more scalable diagnostics.

Methods

The core of this research lies in integrating deep learning with Visual Question Answering (VQA). The authors employed the BLIP (Bootstrapping Language-Image Pre-training) model, a powerful vision-language architecture, and fine-tuned it on their custom wheat rust datasets. A key aspect of their approach is the use of a dual attention mechanism, which allows the model to simultaneously focus on relevant parts of the image and the question.

Furthermore, they implemented a federated learning strategy using the Flower framework. This approach allows multiple clients (e.g., farms) to collaboratively train a central model without sharing their raw data, thus preserving privacy. The central model, a ResNet architecture, was chosen for its superior performance. The federated averaging algorithm was used to aggregate model updates from clients.

Datasets

The study introduces two novel datasets:

  • WheatRustDL2024: This dataset contains 7998 high-resolution images of wheat plants, including healthy leaves and those infected with various types and severities of rust. It's annotated with detailed questions and answers relevant to wheat rust detection, designed to train a VQA system. The dataset captures diverse conditions like varying lighting and obstructions.
  • WheatRustVQA: This is an augmented dataset derived from WheatRustDL2024, comprising 1800 image-text pairs. It was specifically created for training and evaluating the VQA model.

Key Insights and Ideas

  • VQA for Agricultural Diagnostics: The paper demonstrates that VQA is a powerful paradigm for agricultural disease diagnostics, moving beyond simple classification to provide detailed, context-aware information.
  • Federated Learning for Privacy and Scalability: Implementing federated learning is crucial for agricultural applications where data privacy is paramount and for enabling collaborative model improvement across diverse, distributed data sources.
  • BLIP's Effectiveness: The BLIP model, with its strong vision-language capabilities, proved effective for fine-tuning on a specialized agricultural domain like wheat rust detection.
  • Domain-Specific Datasets are Crucial: The creation of the WheatRustDL2024 and WheatRustVQA datasets was essential for training a robust and generalized model that can handle real-world variations.
  • Lightweight and Deployable: The developed federated model is designed to be lightweight, making it suitable for integration into mobile phones and drones for on-field, real-time analysis.

Key Results

  • The fine-tuned ResNet model achieved an accuracy of 97.69% on the existing data for wheat rust classification.
  • The VQA model achieved an average BLEU score of 0.6235 on its testing partition, indicating good performance in generating relevant answers.
  • The federated learning approach, combined with the BLIP model and custom datasets, significantly improved diagnostic accuracy and reduced the need for expert intervention.

Limitations and Future Work

  • Generalizability: While the dataset aims for generalization by including diverse conditions, the models are currently specific to wheat rust. Future work should focus on expanding the dataset to include other crop diseases and exploring multi-disease models.
  • Field Validation: Further validation of the system in varied real-world field conditions is needed.
  • Integration: Integrating the technology into mobile applications for real-time field use is a key future direction.
  • Federated Learning Strategies: Exploring alternative federated learning strategies beyond FedAvg and different network topologies could further enhance performance and robustness.
  • Dataset Size: The WheatRustVQA dataset was noted as "narrow" for fine-tuning, suggesting that larger, more diverse VQA datasets for agricultural applications would be beneficial.

Summary updated

Abstract

This paper presents a novel approach to agricultural disease diagnostics through the integration of Deep Learning (DL) techniques with Visual Question Answering (VQA) systems, specifically targeting the detection of wheat rust. Wheat rust is a pervasive and destructive disease that significantly impacts wheat production worldwide. Traditional diagnostic methods often require expert knowledge and time-consuming processes, making rapid and accurate detection challenging. We drafted a new, WheatRustDL2024 dataset (7998 images of healthy and infected leaves) specifically designed for VQA in the context of wheat rust detection and utilized it to retrieve the initial weights on the federated learning server. This dataset comprises high-resolution images of wheat plants, annotated with detailed questions and answers pertaining to the presence, type, and severity of rust infections. Our dataset also contains images collected from various sources and successfully highlights a wide range of conditions (different lighting, obstructions in the image, etc.) in which a wheat image may be taken, therefore making a generalized universally applicable model. The trained model was federated using Flower. Following extensive analysis, the chosen central model was ResNet. Our fine-tuned ResNet achieved an accuracy of 97.69% on the existing data. We also implemented the BLIP (Bootstrapping Language-Image Pre-training) methods that enable the model to understand complex visual and textual inputs, thereby improving the accuracy and relevance of the generated answers. The dual attention mechanism, combined with BLIP techniques, allows the model to simultaneously focus on relevant image regions and pertinent parts of the questions. We also created a custom dataset (WheatRustVQA) with our augmented dataset containing 1800 augmented images and their associated question-answer pairs. The model fetches an answer with an average BLEU score of 0.6235 on our testing partition of the dataset. This federated model is lightweight and can be seamlessly integrated into mobile phones, drones, etc. without any hardware requirement. Our results indicate that integrating deep learning with VQA for agricultural disease diagnostics not only accelerates the detection process but also reduces dependency on human experts, making it a valuable tool for farmers and agricultural professionals. This approach holds promise for broader applications in plant pathology and precision agriculture and can consequently address food security issues.
BibTeX
@inproceedings{nanavaty2024integrating,
  author = {Akash Nanavaty and Rishikesh Sharma and Bhuman Pandita and Omesh Goyal and Srinivas Rallapalli and Murari Mandal and Vaibhav Kumar Singh and Pratik Narang and Vinay Chamola},
  title = {{Integrating deep learning for visual question answering in Agricultural Disease Diagnostics: Case Study of Wheat Rust}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-79793-2},
}
Figure