An LLM-Orchestrated Multi-Modal Chatbot for Tomato Disease Consultant

Ngoc-Son Le, Khanh-Nghi Phan, Ngoc Hong TranarXiv 2026 · arXiv

TL;DR

Researchers have developed a multi-modal chatbot that uses Retrieval-Augmented Generation (RAG) and local Large Language Models (LLMs) to diagnose tomato diseases from natural language descriptions. By orchestrating Mistral and Llama 3.2 through a lightweight, offline-capable framework, the system provides accessible agricultural expertise to farmers in remote areas where technical support is scarce.

What Problem Does This Paper Address?

Global agricultural productivity is under constant threat from plant pathogens, and tomatoes are notoriously vulnerable to a wide array of pests and physiological disorders. While traditional diagnosis relies on expert inspection or laboratory testing, these resources are often too expensive or geographically distant for smallholder farmers.

Existing AI solutions have largely focused on image classification using Convolutional Neural Networks (CNNs). While accurate, these "black box" models often fail to provide the conversational context or actionable treatment advice that a farmer actually needs. The authors aim to bridge this gap by creating an interactive "consultant" that doesn't just identify a disease but explains it and suggests remedies through a natural language interface.

Methods

The system employs a Retrieval-Augmented Generation (RAG) architecture, which allows the LLM to "look up" facts from a trusted domain-specific database rather than relying solely on its pre-trained internal knowledge.

Key technical components include:

  • Orchestration: LangChain manages the flow between user input, document retrieval, and LLM prompting.
  • Local Inference: The Ollama framework is used to run models locally, which is crucial for the paper's goal of "offline-friendly" operation in low-connectivity environments.
  • Vector Database: ChromaDB stores embeddings of disease data generated by the MXBAI-Embed-Large model.
  • Multi-modal Feedback: While the primary input is text, the system uses a keyword-matching and vector-fallback mechanism to retrieve and display reference images of the diagnosed disease to help the user validate the result.
Figure 2. System architecture and component interaction in the tomato disease Chatbot

The authors also implemented a hybrid deployment strategy using Google Colab and ngrok to provide GPU-accelerated inference for a local Streamlit-based frontend, balancing hardware constraints with performance.

Datasets

The authors manually curated a custom dataset specifically for this project:

  • Text Dataset: 45 entries in JSON format covering various tomato-related diseases, pests, and disorders. Each entry includes scientific descriptions, symptoms, cultural/chemical treatments, and recommended pesticides.
  • Image Dataset: 45 subfolders (one per disease) containing five representative images each, sourced from academic publications and agricultural institutions like PlantVillage and the University of Minnesota Extension.
Figure 4. Sample JSON entry from text dataset

Key Insights and Ideas

  • The "Lite" Multi-modal Approach: Instead of using a massive, computationally expensive vision-language model (VLM), the system uses text-based RAG to identify a disease label, which then triggers the display of relevant reference images. This makes the system much more deployable on modest hardware.
  • Model Trade-offs: The study highlights a clear performance split between the two tested models. Llama 3.2 demonstrated superior semantic relevance and provided more "verbose" (detailed) explanations, making it better for high-accuracy needs. Conversely, Mistral was faster and more lightweight, particularly for complex queries, making it a better candidate for edge deployment.
  • Offline Resilience: By prioritizing local inference and structured JSON chat histories, the architecture is specifically designed for the "low-connectivity agricultural environments" typical of remote farming.

Key Results

The models were evaluated across five test queries (Q1-Q5) of increasing difficulty.

  • Relevance Score: Llama 3.2 consistently outperformed Mistral, achieving relevance scores between 0.60 and 0.68, whereas Mistral ranged from 0.37 to 0.64.
  • Response Time: Mistral was generally faster on complex queries (e.g., 7.19s vs. 10.74s for Q5). However, Llama 3.2 was surprisingly fast on simpler prompts (4.65s for Q2).
  • Grammar and Fluency: Both models achieved near-perfect grammar scores (0.99 to 1.00), ensuring that the advice provided to farmers is clear and professional.
  • Visual Validation: The system successfully retrieved diagnostic imagery, such as "bird's-eye spots" for Bacterial Canker, providing a visual "aha moment" for the user.
Figure 6. Bacterial Canker affecting mature red tomatoes (symptoms: bird’s-eye spots)

Limitations and Future Work

The current system is a promising prototype, but the authors acknowledge several areas for growth:

  • Dataset Scale: Supporting only 45 diseases is a start, but the authors plan to expand the dataset to cover a wider variety of tomato pathologies and perhaps other crops.
  • Retrieval Accuracy: The "moderate" relevance scores suggest that the RAG pipeline could be further optimized, perhaps through better document chunking or fine-tuning the embedding model.
  • Advanced Multi-modality: Future iterations aim to move beyond simple image retrieval toward more complex multi-modal reasoning, potentially allowing the user to upload a photo of their own plant for direct visual diagnosis by the LLM.

Summary updated

Abstract

This paper introduces a chatbot developed to assist in diagnosing common diseases in tomato plants through natural language interaction. Users can describe symptoms in text. Then, the chatbot responds with the most likely disease, and basic treatment suggestions and related image-based reasoning. The system leverages pre-trained Mistral and Llama 3.2 large language models. The system is also built using LangChain and Streamlit to provide a lightweight and offline-friendly interface. This work targets to support tomato growers a fast and accessible tomato plant health guidance when the technical expertise is not available nearby, especially in the remote areas. The experimental results are promising and encourage further improvement of the chatbot for future real-world deployment.
BibTeX
@misc{le2026llm,
  author = {Ngoc-Son Le and Khanh-Nghi Phan and Ngoc Hong Tran},
  title = {{An LLM-Orchestrated Multi-Modal Chatbot for Tomato Disease Consultant}},
  year = {2026},
  eprint = {2611.10608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10608},
}
Figure