Language-model farm advisors are being built for smallholders who mostly speak regional, non-standard varieties of low-resource languages. These systems are graded by how closely their answers match what a call-center agent wrote that day. Two blind spots meet here. The models are rarely tested on the dialects their users actually speak. And the agent's answer is a short-term reference: it says nothing about whether the advice holds up over many seasons. Long-term field trials show that practices which raise yield now can damage soil over decades. In a 41-year experiment, high-dose chemical fertilizer cut soil macroaggregates (the large crumbs that give soil its structure) by 19% and acidified the soil, while organic regimes nearly doubled soil carbon stocks (Assessing the Impacts of Fertilization Regimes on Soil Aggregate Dynamics in Northeast China).
We propose to measure, and then reduce, a specific failure: when an advisor misunderstands a dialectal query, it may fall back to generic, yield-now advice that ignores long-term soil evidence and local soil conditions. If that is true, the harm from dialectal exclusion is larger than accuracy benchmarks suggest, because it lands on the outcome that takes longest to notice. We test the hypothesis with a horizon-aware evaluation, meaning one that grades advice against long-term trial evidence. We also test a cheap remedy: before generating anything, convert the farmer's query into a small structured description of what they are asking about, using a character-level reader trained mostly on synthetic dialect text.
Core Claim
Our central empirical claim is that the dialectal gap in farm advice is larger when the advice is graded for long-horizon soundness than when it is graded for surface match to agent notes. We define the gap the way DIALECTBENCH does (DIALECTBENCH): the relative drop in score from a standard variety $u$ to a non-standard variety $v$,
$$\text{Gap}_m(v,u) = \frac{m(u) - m(v)}{m(u)},$$
where $m$ is the metric being compared. We compute this gap twice, once with $m$ = answer correctness against the agent note and once with $m$ = a horizon-consistency score, and test whether the second gap is reliably larger.
The horizon-consistency score has two parts. The first asks whether the advice agrees with long-term trial evidence where such evidence exists. The second asks whether the advice flags that the evidence comes from a different soil or climate when it does.
Our methodological claim is narrower. Most of this horizon gap comes from misreading the query, not from the generator lacking knowledge. So a query-understanding step built to tolerate spelling variation should close the horizon gap more than it closes the surface gap. Translating the query into English first should close it less.
This differs from the closest existing work on two axes. AgriLLM fine-tunes Flan-T5 on English transcripts of Kisan Call Center queries and scores against agent answers. It never sees dialectal input and never asks whether the advice is sustainable. My Climate Advisor retrieves from scientific literature using hybrid keyword and embedding search, and is graded by experts on seven criteria including scientific accuracy. It assumes standard English queries, and its rubric does not distinguish evidence from two-season trials from evidence from multi-decade trials. Our work adds both the dialect axis and the time-horizon axis, and asks how they interact.
Research Program
Horizon-tagged evidence store. We start from the agricultural literature corpus used by My Climate Advisor (S2ORC plus the top agricultural journals). We keep field trials and tag each extracted claim with three things: trial duration, soil type and site, and the outcome measured (yield, nitrogen loss, soil carbon, pH, aggregate structure).
The trials in the research file anchor the tagging guidelines. The 41-year dark brown soil experiment is the model of a decadal claim. The two-year slow-release nitrogen blend study is the model of a short-horizon claim. That study cut apparent nitrogen loss by 43–62% and recommended a 53–58% slow-release ratio, but its authors say it cannot address long-term soil effects (Blending of Slow-Release N Fertilizer and Urea…).
Pedological specificity is a real constraint. Chinese dark brown soil results do not transfer to Tamil Nadu. For that reason, a sound answer is sometimes not a recommendation but a statement that the long-term evidence is non-local.
Horizon-consistency rubric. Agronomists will label advisor outputs on two questions. First, does the advice contradict long-term evidence relevant to the query? Recommending sustained high-dose chemical nitrogen with no acidification caveat is the clearest example. Second, when only short-horizon or non-local evidence exists, does the advice say so?
The rubric is graded against the evidence store, not against an agent note. This is how we stand in for long-term impact assessment that no NLP project can run directly. It mirrors the way An Adoption-Aware Crop Recommender from Farmer World Feedback fixes part of its model to coefficients from agronomic literature rather than learning everything from noisy observational data. That paper motivates the anchoring idea. Its own target, single-season realized yield, is exactly the short horizon we want to move beyond.
Dialectal query sets. We build two kinds of test queries from a sample of KCC English queries.
The first is a controlled synthetic probe. We apply Multi-VALUE's rule-based rewrites to produce synthetic Indian, Singapore and other English varieties (Multi-VALUE). This isolates grammatical variation with the meaning held fixed. It matters because Multi-VALUE showed BERT's conversational question-answering accuracy falls 25.4% on Singapore English.
The second is a real elicited set. Speakers of several regional Tamil varieties re-ask the same queries in their own words and spelling, as text and as speech transcribed by a speech recognizer. This set carries the lexical and spelling variation that Multi-VALUE cannot generate. Each elicited query keeps the gold agent answer and gold intent of its English source, so the meaning is controlled across varieties.
Remedy: a structured intent step. We do not fine-tune the answer generator on scarce dialect data. DIALECTBENCH's "fine-tuning paradox" constrains this design: tuning on small in-variety sets often widened the gap because of noise. Instead, we train only a small character-level reader that maps a query to a fixed set of slots: crop, growth stage, soil or site description, input under discussion, problem, and the time horizon the farmer is asking about.
The method comes from character-level dialect work, which handles spelling noise by reading character sequences rather than whole words. Examples are the character-level convolutional and recurrent network for German dialects (Mohamed Ali) and character n-gram classifiers for Arabic dialects (Eltanbouly et al.; Meftouh et al.).
Training data comes from three sources: synthetic dialect rewrites, spelling-perturbed KCC queries, and a few hundred elicited queries labeled by active learning, following Active Learning for Multidialectal Arabic POS Tagging. The slots then drive retrieval from the horizon-tagged store, and an unchanged generator writes the answer.
Baselines
The naive reference point is the AgriLLM recipe: Flan-T5 Base fine-tuned on KCC English transcripts, run directly on dialectal queries. AgriLLM reports BLEU 0.555 and F1 0.837 overall, and BLEU 0.628 on fertilizer queries (AgriLLM). We retrain it on our KCC split so the numbers are comparable. Beating it means a lower horizon gap at equal or better correctness on standard queries.
The strongest existing system is a My Climate Advisor-style pipeline: hybrid keyword and embedding search over the same evidence store, with Llama 3-8B and greedy decoding (My Climate Advisor). It is the strongest competitor because it already grounds advice in science rather than agent notes. No quantitative result was found on Lacuna for My Climate Advisor; the paper reports only that it was comparable to GPT-4 on accuracy and provenance. We give it the same evidence store and the same generator as our system, so the only difference is how the query is read. Beating it means a statistically smaller horizon gap on the elicited Tamil set.
Translate-then-retrieve is the third baseline. It normalizes queries with NLLB into English before the same retrieval and generation. NLLB is one of the models DIALECTBENCH evaluates.
The fourth baseline is a frontier model prompted zero-shot to consider long-term soil health, with no retrieval.
Ablations isolate the key ingredients:
- Replace the intent step with plain query text, keeping horizon tags.
- Keep the intent step but drop the horizon tags.
- Train the intent reader without synthetic dialect data.
- Fine-tune the generator end-to-end on the same small dialect budget. This tests the fine-tuning paradox in this domain.
Every trained component gets the same labeled-data budget, the same tuning trials, and the same evaluation-set isolation.
Why not just…?
Why not just translate dialect queries into English and use a standard advisor? This is the most serious threat, and it is included as a baseline rather than argued away. DIALECTBENCH shows NLLB and similar models degrade sharply on non-standard varieties, and phonetic spelling is a known failure mode. But translation could still recover the slots that matter for horizon soundness, such as crop and input, even when the wording comes out wrong. If translate-then-retrieve matches our system's horizon gap on the elicited set, the intent step is unnecessary and we will report that.
Why not just prompt a large model to think about long-term soil health? A prompt can add caveats. It cannot tell a two-season result from a 41-year one, and it cannot recognize that evidence is non-local unless it already knows the trial literature. The AfriSenti results suggest general models also misread unrepresented varieties: zero-shot transfer to unseen languages topped out at 58.15 average F1 (AfriSenti-SemEval). The prompted baseline tests this directly. If it closes the horizon gap without retrieval, the evidence store adds nothing for this purpose.
Why not just fine-tune the advisor on dialectal call transcripts? DIALECTBENCH found that in-variety fine-tuning on small, noisy sets often widened the gap, and we expect the same here. Ablation 4 trains the generator end-to-end on the identical dialect budget, so the experiment shows whether the paradox holds in this domain or whether simple fine-tuning suffices.
Proposed Experiments and Decisive Checks
The first check validates the metric. On standard English KCC queries, we correlate BLEU against agent notes with the horizon-consistency score across all systems and queries. A weak or negative correlation would show that agent-note scoring, the current evaluation practice, misses the dimension we care about. A strong correlation would make the second metric redundant and weaken the core claim.
The second check tests the central hypothesis. We compute $\text{Gap}_m$ for correctness and for horizon consistency on the synthetic English varieties and on the elicited Tamil varieties, with bootstrap confidence intervals over queries. The claim holds only if the horizon gap is reliably larger than the correctness gap. We also inspect the failures: are dialect-induced errors disproportionately generic high-input recommendations, or wrong-crop answers?
The third check tests the remedy. It holds if the intent step reduces the horizon gap more than translate-then-retrieve does at matched correctness, and if slot accuracy on elicited queries predicts horizon consistency better than translation quality predicts it.
The fourth check is about honesty. We report, per query category, how much of the evidence store is local, multi-season, or absent. Where long-term evidence simply does not exist for a region, the right output of this project is an annotated hole in the evidence, not a better-sounding answer.
References
- Assessing the Impacts of Fertilization Regimes on Soil Aggregate Dynamics in Northeast China
- DIALECTBENCH: An NLP Benchmark for Dialects, Varieties, and Closely-Related Languages
- AgriLLM: Harnessing Transformers for Farmer Queries
- My Climate Advisor: An Application of NLP in Climate Adaptation for Agriculture
- Blending of Slow-Release N Fertilizer and Urea Improve Rainfed Maize Yield and Nitrogen Use Efficiency While Reducing Apparent N Losses
- An Adoption-Aware Crop Recommender from Farmer World Feedback
- Multi-VALUE: A Framework for Cross-Dialectal English NLP
- Character Level Convolutional Neural Network for German Dialect Identification
- Simple But Not Naïve: Fine-Grained Arabic Dialect Identification Using Only N-Grams
- The SMarT Classifier for Arabic Fine-Grained Dialect Identification
- Active Learning for Multidialectal Arabic POS Tagging
- SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)