Retrieval-augmented generation (RAG) lets a language model answer by first fetching passages from a document collection and then writing an answer grounded in them. For low-resource languages, building such a system means making many choices at once. Should the question be machine-translated to English before searching? Should the search use keyword matching, a dense multilingual embedding, or both? Which generator should write the answer, and how many passages should it read? Each choice trades answer quality against cost and latency, and the best choice seems to differ by language. The standard tool for this is multi-objective model selection, which maps the Pareto frontier: the set of configurations where no objective can improve without another getting worse. It assumes that each candidate's objectives can be measured reliably. In low-resource languages they usually cannot. The labeled questions available for choosing a pipeline often number in the tens or low hundreds per language. With that few, the configuration that looks best is often just the luckiest.
This note argues that low-resource RAG is limited mainly by how little evaluation data exists per language, and much less by the cost of searching over configurations. We propose to select pipelines with a hierarchical statistical model. It pools evidence across languages that share script, family, or resource level, and it breaks each pipeline into its components so that evidence about one configuration informs estimates for related ones. The goal is concrete: choose, from about fifty labeled questions per language, cost–quality trade-offs whose true held-out performance comes close to what one would choose with thousands of questions. Where the selection is uncertain, the method should also say where a few more labels would help most.
Core Claim
The claim has three testable parts. The first is that naive per-language Pareto selection on small validation sets suffers a large winner's curse: the chosen configurations do worse on held-out data than their validation scores promised, and they are often dominated on held-out data. A back-of-envelope calculation shows why. With $n$ questions, the standard error of an accuracy-like score near 0.3 is about $\sqrt{0.3 \cdot 0.7 / n}$, roughly 0.065 at $n=50$. That is comparable to or larger than many differences between competing pipelines. The second is that partial pooling across languages and across pipeline components removes most of that loss. The third is that choosing a single configuration for all languages, which is what practitioners actually do, is also measurably worse, because the best bridging strategy differs by language.
The ingredient that is new here is the selection estimator. We do not claim a new retriever, translator, or search algorithm.
Relation to Prior Work
The closest work is Faster, Cheaper, Better. It runs multi-objective Bayesian optimization with noisy expected-hypervolume acquisition over RAG hyperparameters (embedding model, chunking, number of retrieved chunks, generator) against faithfulness, helpfulness, cost, and latency on two English benchmarks. It supplies the method we reuse for proposing configurations and the evaluation measure we use, the hypervolume of the selected set. Its benchmarks already have only 50 and 86 questions. Its authors treat noise through the acquisition function, but they do not ask whether the selected front holds up on fresh data. They do report that optimal configurations differ between tasks. We treat languages the way they treat tasks, and we make the noise in measuring the frontier itself the object of study.
Retriever Portfolios picks a small set of complementary retrievers by greedy submodular maximization of best-of-$k$ coverage. It shows on English multi-hop QA that two diverse retrievers fetching four documents each beat one retriever fetching twenty. It motivates including diverse bridging routes in our candidate pool. It optimizes accuracy only, and it assumes a query sample large enough to estimate coverage, which is the assumption we drop.
Four papers establish that the pipeline trade-offs are real and depend on the language, without optimizing over them. XOR QA builds a benchmark that queries English Wikipedia from seven languages. It finds that translate-then-retrieve beats end-to-end multilingual retrieval but trails human translation. AfriQA extends this to ten African languages and finds that direct mDPR retrieval on native queries collapses while machine translation plus mDPR works best end to end. PARC retrieves English examples by semantic similarity without translation and finds that gains track each language's closeness to English. TraSe routes Bangla questions through English inside the prompt and has an LLM choose among five prompting strategies. It does this on 200 questions and ignores the five-fold cost.
On the other side, AfriMTEB and AfriE5 shows that a compact embedding model adapted with cross-lingual contrastive distillation reaches 63.7 on AfriMTEB-Lite. That makes direct, translation-free retrieval a live and cheap option. AfriqueLLM shows that continued pretraining gives AfriqueQwen-14B a 63.58 AfroBench macro score against 60.71 for Gemma 3 27B, so model size is a poor proxy for quality in these languages.
Together these papers leave open what our work tests. The known contradiction, where translation is required in AfriQA but unnecessary in PARC and AfriE5, may not be a contradiction at all. The right bridge may simply differ by language, and nobody has shown how to find it for each language from realistic amounts of data.
Research Program
We first build a factored configuration space. Query bridging is one of none, NLLB, a commercial MT system, or translation inside the LLM prompt as in TraSe. Retrieval is BM25, mDPR, hybrid BM25 plus mDPR, or AfriE5 (mE5 for the XOR languages). Reranking with BGE Reranker v2 m3 is on or off. Retrieval depth ranges over a small grid. The generator is Qwen 3 8B, AfriqueQwen-8B, AfriqueQwen-14B, or Gemma 3 27B. The answer is either written directly in the target language or written in English and translated back. For every configuration and language we record three objectives: answer F1 in the target language, cost measured as GPU-seconds plus API tokens priced at fixed rates, and median end-to-end latency on fixed hardware. As an equity objective we add the worst-language F1 across the deployment set.
The selection model treats the F1 of configuration $c$ on language $\ell$ as
$$ y_{c,\ell} = \sum_{j} \beta_{j}(c) + u_{g(\ell)}(c) + v_{\ell}(c) + \epsilon, $$
where $\beta_j(c)$ is the shared effect of the component $c$ uses in slot $j$ (for example, "hybrid retrieval"). The term $u_{g(\ell)}(c)$ is a deviation for the language group $g(\ell)$, defined by script and family and relating to $c$ through its components. The term $v_\ell(c)$ is a language-specific deviation, shrunk toward zero with a variance learned from the data. The noise $\epsilon$ has binomial-scale variance set by the number of labeled questions. Cost and latency are cheap to measure without labels, so we observe them directly on unlabeled queries rather than model them.
Because of the shrinkage, the model interpolates between two extremes. When languages disagree strongly and there is enough data, each language gets its own choice. When evidence is thin, the model falls back to the best configuration for the group or for all languages.
Selection works on posterior samples of $y$. For each language we report configurations whose posterior probability of being non-dominated on (F1, cost, latency) exceeds a threshold. For a stated budget, we deploy the one with the highest expected F1 among those that satisfy the budget with high probability. Candidate configurations to label are proposed with the noisy hypervolume acquisition of Faster, Cheaper, Better, applied to the pooled posterior. A second acquisition rule decides which language would gain most from additional labeled questions. That rule speaks to the practical question of where a small annotation budget should be spent.
Baselines
The naive reference is fixed-heuristic deployment with no selection. For XOR-TyDi QA we use the pipeline that XOR QA reports as its best on the full round-trip task, which reaches an average F1 of 18.7. For AfriQA we use the combination of commercial MT and mDPR that AfriQA reports as the strongest automated end-to-end system; the file does not give that system's numeric F1. Its reported 73.9% recall@10 for hybrid retrieval over human-translated queries serves as an upper reference for retrieval. We rerun both with our generators so that cost and latency are measured on identical hardware. The selection methods must beat this on held-out F1 at equal or lower cost. Otherwise selection is not worth doing.
The strongest existing method is per-language multi-objective Bayesian optimization as in Faster, Cheaper, Better. It has noisy hypervolume acquisition, is run independently for each language on that language's labeled questions, and receives exactly the same number of pipeline evaluations and labeled questions as our method. The file reports no numeric hypervolume for it; its gains are shown only graphically. It is the strongest competitor because it already handles noisy objectives and has beaten Sobol, uniform, and standard expected-hypervolume search. Our method beats it if the configurations it selects achieve higher held-out hypervolume and smaller winner's-curse gaps at $n \le 100$ questions per language, with confidence intervals over repeated subsamples that exclude zero difference.
The second strong competitor is fully pooled selection: the same optimizer run on the union of all languages' questions to pick one set of trade-offs for everyone. This is the most common real-world practice, and the practitioner survey notes that practitioners tend toward simple grid or random search. We run a pooled grid search as well. The comparison is fair because all methods see the same labeled questions.
Three ablations isolate our key ingredient. One removes the component factorization, so each configuration gets its own intercept. One removes the language-group level, so there is only a global mean and per-language deviations. One replaces typology-based groups with random groups. If random groups perform as well as typology-based ones, the gain comes from shrinkage alone and not from linguistic structure.
Why not just…?
Why not just use the best-known translate-then-retrieve pipeline everywhere? Both XOR QA and AfriQA found machine translation followed by dense retrieval to be the strongest automated option, so this is a serious default. Those studies, however, did not include adapted embeddings like AfriE5, did not count the cost of translation, and did not report whether the ranking holds in every language. PARC finds that retrieval quality depends on how close a language is to English. The comparison against our naive baseline settles the question. If the translation pipeline is on or near the held-out Pareto front for every language, the per-language diversity we assume is not there, and we will report that.
Why not just select one configuration on all languages pooled together? This alternative is a real threat. If one pipeline dominates across languages, pooling is the lowest-variance estimator and our hierarchical model can at best match it. We do not argue around this. The model reduces to pooling when the learned variance across languages is near zero, so the experiment measures that variance directly. Our claim survives only if the fully pooled baseline loses held-out F1 on a clear subset of languages, which we would expect for languages written in non-Latin scripts or unseen by the base models, given PARC's and AfriQA's findings.
Why not just run the existing multi-objective optimizer separately in each language? Search is not what runs short here; measurement is. The benchmarks in Faster, Cheaper, Better are already small, and at $n=50$ the sampling error in F1 is on the order of the differences between pipelines. The optimizer's noise handling keeps its search from being misled, but it cannot make up for sampling error in the final choice. The winner's-curse measurement in our experiments tests this directly.
Why not just prompt a bigger model with no retrieval? XOR QA shows that retrieving from English raised answer coverage by 22% on average and by 40% for Bengali. TraSe reports 22% accuracy without its retrieval-grounded method against 34% with it. We nonetheless include "no retrieval" configurations for each generator in the search space, so this option is tested rather than assumed away.
Proposed Experiments
The main experiment uses AfriQA (ten languages, over 12,000 examples) and XOR-TyDi QA (seven languages) to simulate scarcity with ground truth. For each language we draw a selection set of $n \in \{25, 50, 100, 200\}$ labeled questions and keep the rest as an oracle test set, large enough that its objective estimates are nearly noise-free. We repeat this over twenty random draws. Every method selects configurations from the selection set only. We then score its choices on the oracle set with four measures. The first is held-out hypervolume of the selected set, normalized by the oracle front's hypervolume. The second is regret: the F1 lost at a fixed cost budget relative to the oracle-best configuration within that budget. The third is the winner's-curse gap, the promised F1 minus the held-out F1. The fourth is the fraction of selected configurations that turn out to be dominated on the oracle set.
The decisive result would show three things at $n=50$. Per-language Bayesian optimization has a substantial winner's-curse gap. Full pooling has high regret on an identifiable subset of languages. Our model recovers most of the oracle hypervolume in both cases. All three must hold in both benchmarks for the core claim to stand.
Two further checks probe the extremes. In leave-one-language-out selection, we choose a configuration for a language using no labels of its own, drawing only on its group and on unlabeled cost and latency measurements. This tests whether the approach gives anything for languages with zero evaluation data. On the Bangla dataset from TraSe, where only 200 labeled questions exist, we check whether the model's choice, informed by XOR's Bengali data, beats TraSe's reported 34% accuracy under automatic retrieval at a fraction of its five-candidate cost.
Finally, we test the label-allocation rule. With a fixed extra budget of labeled questions, we compare spending it where the model directs against spreading it uniformly across languages, measured in held-out hypervolume gained per label.
A null result would also be informative. If shrinkage estimates near-zero variation between languages and full pooling matches everything else, then practitioners in these languages can choose one pipeline from pooled data. That finding would narrow the gap described in the file to a much simpler recommendation.
References
- Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems
- Retriever Portfolios: A Principled Approach to Adaptive RAG
- XOR QA: Cross-lingual Open-Retrieval Question Answering
- AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages
- Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages
- Empowering Low-Resource Languages: TraSe Architecture for Enhanced Retrieval-Augmented Generation in Bangla
- AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages
- AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
- Practitioner Motives to Use Different Hyperparameter Optimization Methods