Cross-Modal Data Fusion via Vision-Language Model for Crop Disease Recognition

Wenjie Liu, Guoqing Wu, Han Wang, Fuji RenSensors 2025 · OpenReview

TL;DR

This paper introduces a novel cross-modal approach for crop disease recognition by fusing visual leaf images with AI-generated textual descriptions. By leveraging a vision-language model, their method achieves state-of-the-art accuracy with significantly fewer parameters, making it suitable for resource-constrained environments.

What Problem Does This Paper Address?

Crop diseases are a major threat to global food security, leading to substantial yield losses. Early and accurate identification is crucial for effective management. While deep learning, particularly Convolutional Neural Networks (CNNs), has shown promise in image-based disease detection, these methods often overlook valuable contextual information that can be provided by textual descriptions of disease symptoms. Acquiring large, labeled datasets for all possible diseases and crop varieties is challenging, and existing models can struggle with variations in sensor data and environmental conditions. This paper aims to bridge this gap by integrating textual symptom descriptions with visual data to improve the robustness and accuracy of crop disease recognition systems.

Methods

The proposed framework, Cross-Modal Data Fusion via Vision-Language Model (CMDF-VLM), consists of three main stages:

  1. Textual Description Generation: For each input crop leaf image, an advanced image comprehension model (Zhipu.AI's GLM-4V-Plus) is used to generate detailed textual descriptions. These descriptions are structured into three categories:
  2. Global Description: Summarizes the overall health status and disease distribution pattern.
  3. Local Lesion Description: Details the shape, size, edge characteristics, and surface features of individual lesions.
  4. Color-Texture Description: Characterizes the chromatic abnormalities and textural properties of the affected areas. These generated texts are then encoded into semantic feature vectors using a pre-trained text encoder (BLIP2).

  5. Cross-Modal Feature Fusion: This stage involves an image encoder to extract visual features from the input image. These visual features are then iteratively fused with the textual features through multiple cascaded "feature fusion modules." Each module incorporates a Restormer block (an efficient Transformer variant) and a cross-attention mechanism. The cross-attention module uses textual features as a condition to guide the extraction of relevant visual features, effectively aligning the two modalities and prioritizing disease-specific regions.

  6. Classification Prediction: The fused multimodal features are then passed through a final classification module to predict the probability of different disease classes.

A key aspect of their approach is that the vision-language model and text encoder are frozen during training, meaning only the parameters within the feature fusion and classification modules are learned. This significantly reduces the number of trainable parameters.

Datasets

The CMDF-VLM framework was evaluated on three agricultural disease datasets:

  • Soybean Disease Dataset: Contains 10,722 image-text pairs across 8 disease classes.
  • AI Challenge 2018 Dataset: A larger dataset with 35,000 image-text pairs covering 59 categories (10 crop types, 27 disease types).
  • Plant Village Dataset: A comprehensive dataset with 54,305 image-text pairs across 36 classes (14 crop species).

For these datasets, the authors generated hierarchical textual descriptions using the GLM-4V-Plus model, ensuring each description included the global, local lesion, and color-texture components. For healthy leaves, a standardized description was used.

Key Insights and Ideas

  • Textual descriptions are powerful complements to visual data: The study strongly suggests that detailed textual descriptions of crop diseases, even when generated by an AI, can significantly enhance the accuracy and robustness of disease recognition models beyond what is achievable with images alone.
  • Cross-modal attention is key for alignment: The use of cross-attention mechanisms effectively bridges the gap between visual and textual modalities, allowing the model to focus on the most diagnostically relevant information from both sources.
  • Leveraging pre-trained vision-language models is efficient: By using frozen, pre-trained models for text generation and encoding, the authors achieve high performance with a remarkably small number of trainable parameters. This makes the model more practical for deployment.
  • Hierarchical text features are beneficial: Breaking down the textual description into global, local, and color-texture components allows for a more nuanced understanding of disease characteristics, leading to improved performance.

Key Results

  • State-of-the-art accuracy: CMDF-VLM achieved superior accuracy compared to image-only methods on all three datasets. Specifically, it reached 98.74% on the Soybean Disease dataset, 87.64% on the AI Challenge 2018 dataset, and 99.08% on the Plant Village dataset.
  • Parameter efficiency: The model boasts a significantly reduced parameter count, with only 1.14 million trainable parameters. This is notably fewer than many other state-of-the-art models, including ShuffleNet-v2 (4.72M parameters) while achieving higher accuracy.
  • Ablation studies confirm component importance: Adding textual features incrementally (global, then local, then color-texture) consistently improved accuracy, demonstrating the value of each descriptive component. The full model incorporating all textual features yielded the best results.
  • Visualizations support attention: Grad-CAM visualizations showed that CMDF-VLM highlighted more extensive and relevant areas in the images compared to baseline models like ResNet-50 and ShuffleNet-v2, indicating more effective attention to disease-specific features.

Limitations and Future Work

  • Deployment latency: The authors acknowledge that passing images through advanced image comprehension models for description generation during deployment might introduce a slight delay, although they argue the overall benefits outweigh this.
  • Real-time deployment and few-shot learning: Future work will focus on exploring real-time deployment strategies and developing methods for few-shot adaptation to address rare diseases, which are often challenging to collect data for.
  • Generalizability to other crops/diseases: While tested on multiple datasets, further validation across a wider range of crops and diseases would be beneficial.

Summary updated

Abstract

Crop diseases pose a significant threat to agricultural productivity and global food security. Timely and accurate disease identification is crucial for improving crop yield and quality. While most existing deep learning-based methods focus primarily on image datasets for disease recognition, they often overlook the complementary role of textual features in enhancing visual understanding. To address this problem, we proposed a cross-modal data fusion via a vision-language model for crop disease recognition. Our approach leverages the Zhipu.ai multi-model to generate comprehensive textual descriptions of crop leaf diseases, including global description, local lesion description, and color-texture description. These descriptions are encoded into feature vectors, while an image encoder extracts image features. A cross-attention mechanism then iteratively fuses multimodal features across multiple layers, and a classification prediction module generates classification probabilities. Extensive experiments on the Soybean Disease, AI Challenge 2018, and PlantVillage datasets demonstrate that our method outperforms state-of-the-art image-only approaches with higher accuracy and fewer parameters. Specifically, with only 1.14M model parameters, our model achieves a 98.74%, 87.64% and 99.08% recognition accuracy on the three datasets, respectively. The results highlight the effectiveness of cross-modal learning in leveraging both visual and textual cues for precise and efficient disease recognition, offering a scalable solution for crop disease recognition.
BibTeX
@inproceedings{liu2025cross,
  author = {Wenjie Liu and Guoqing Wu and Han Wang and Fuji Ren},
  title = {{Cross-Modal Data Fusion via Vision-Language Model for Crop Disease Recognition}},
  booktitle = {Sensors},
  year = {2025},
  url = {https://openreview.net/forum?id=9RZjFRBLP2},
}
Figure