TL;DR
This paper introduces "My Climate Advisor," a novel question-answering system that synthesizes climate adaptation information from scientific and grey literature for farmers and advisors. It addresses the critical need for accessible, trustworthy climate information in agriculture by developing a Retrieval Augmented Generation (RAG) system and a new evaluation framework tailored for this domain.
What Problem Does This Paper Address?
Climate change poses significant threats to agriculture, necessitating adaptation strategies to ensure resilience and food security. However, accessing and understanding relevant, up-to-date information on climate impacts and adaptation practices can be challenging for farmers and agricultural advisors. This information is often scattered across diverse sources, including peer-reviewed literature and industry reports, making it difficult to synthesize into actionable insights. Existing tools may lack the specificity or trustworthiness required for this domain. The authors aim to bridge this gap by creating a system that can intelligently answer user questions by drawing from a curated corpus of relevant literature.
Methods
The core of "My Climate Advisor" is a Retrieval Augmented Generation (RAG) system. This approach combines a retrieval component to find relevant information from a corpus with a generative language model (LLM) to synthesize an answer.
- Data Indexing: The system indexes three corpora:
- A large corpus of general agricultural and environmental science articles from S2ORC.
- A more specialized corpus of top-ranked agriculture journals.
-
A curated collection of Australian climate risk and adaptation information, including books and industry reports. Documents are chunked into manageable sizes and embedded using a sentence encoder (JinaBERT) to create contextual embeddings.
-
Retrieval: A hybrid scoring mechanism is used, combining keyword matching (BM25) with semantic similarity (JinaBERT embeddings) to retrieve relevant document chunks.
- Generation: An open-source LLM, Llama 3-8b, is used for answer generation. The system employs instruction fine-tuning and a greedy decoding strategy to ensure reproducibility and reduce hallucinations. The use of open-source models is a deliberate choice to ensure data privacy and intellectual property protection.
- Evaluation Framework: Recognizing that standard QA evaluation metrics are insufficient for this domain, the authors developed a novel framework with seven criteria (Context, Readability, Language, Provenance, Specificity, Comprehensiveness, Scientific Accuracy) and an annotation guideline.
Datasets
- S2ORC Corpus: A large corpus of scientific articles, filtered for "Agricultural and Food Sciences" and "Environmental Science."
- Top Agriculture Journals: A subset of documents from the top 100 agriculture journals, supplemented from Elsevier.
- Australian Climate Risk and Adaptation Data: Expert-curated documents, books, and industry reports specific to Australian agriculture.
- My Climate View API: Used for location and commodity-specific climate projection data.
No new benchmark datasets were introduced; instead, the authors created a set of 15 questions curated by domain experts for evaluation.
Key Insights and Ideas
- Open-Source LLMs for Domain-Specific Tasks: The study demonstrates that open-source LLMs, when combined with RAG and domain-specific data, can perform competitively with proprietary models, offering significant advantages in data privacy and cost.
- RAG for Scientific Accuracy and Provenance: The RAG approach significantly enhances the scientific accuracy and provenance of the generated answers, a critical factor for trustworthy advice in agriculture.
- Domain-Specific Evaluation is Crucial: Standard QA metrics are inadequate for evaluating climate adaptation advice. The authors' custom evaluation framework, developed with domain experts, provides a more nuanced assessment of response quality.
- Model Scale vs. Specificity: While larger models generally perform better, the study found that smaller models, when not strictly adhering to RAG, sometimes scored higher by relying on their internal knowledge, highlighting a trade-off between instruction-following and general knowledge recall.
Key Results
- Competitive Performance: While GPT-4 was generally preferred across most metrics, the authors' RAG-based Llama 3 model performed comparably, especially in scientific accuracy and provenance.
- Annotator Agreement: Despite initial disagreements on scientific accuracy, annotator agreement was moderate to strong when considering overall sentiment, indicating a general alignment in judging response quality.
- Question Difficulty: Questions requiring more reasoning or specific local context proved more challenging for LLMs than knowledge-recall questions.
- RAG vs. Non-RAG: The RAG models were generally more scientifically accurate and provided better provenance, though sometimes at the expense of readability and context.
Limitations and Future Work
- Prompt Engineering: The study used default settings for most models, with temperature set to 0. Future work could explore more sophisticated prompt engineering to improve readability and context.
- Australian Context: Some questions were not explicitly Australian-contextualized, leading to some models providing global case studies, which impacted specificity scores. Future work could refine question phrasing.
- User Interface: The current interface is in early development and primarily for feedback collection. Future improvements will focus on enhancing retrieval and generation capabilities.
- Multimodal Input: The authors plan to expand input capabilities to include numerical data and graphs for more comprehensive climate data representation.
- Fine-tuning for Readability: The RAG system's output, while accurate, can be dense. Future work will focus on fine-tuning the model to improve readability and structure, potentially by including references at the end of the answer.
Summary updated