Passage-Anchored Native Benchmarks for Open-Book Cultural Evaluation

Benchmarks built from real national exams, such as Indian competitive exams, Bangladeshi textbooks, and Italian civil-service tests, keep reporting that language models fail on local law, history and civics. MILU, for example, finds some India-specific subjects near random chance. Every one of these benchmarks tests models closed-book, so they cannot say why the model fails. It may never have seen the fact. It may be unable to read the native-language document that contains it. Or the question may be badly built. We propose a different way to build such benchmarks. Each question is generated from a specific passage of a native-language source document. That passage is kept as gold evidence, and the source documents become the search collection for open-book testing. A cheap open model drafts the questions, and native-speaker reviewers spend their limited time only on the drafts that automatic checks cannot settle.

The opportunity is that one construction step yields two things at once. The first is a cost-accounted recipe for growing native benchmarks without falling back on machine translation. The second is a benchmark that separates "didn't know" from "couldn't find" from "couldn't read." That separation is the missing diagnostic behind every cultural-gap number in the literature.

Core Claim

We claim two things. First, cultural-knowledge questions anchored to native passages and verified through triaged human review can be produced at a small fraction of the human time of manual authoring. Triaged review means people review only the items that automatic checks flag as uncertain. Their error rate should be comparable to manual authoring, and they should rank models the same way real native exam questions do. Second, running these items closed-book, with the gold passage supplied, and with real end-to-end retrieval will show how much of the documented cultural gap disappears once the model can look facts up in native text. The second claim is the scientific payoff, and the first is what makes it affordable across languages.

What is new is not "add retrieval to a cultural benchmark." Anyone can attach a search engine to existing exam items. What is new is building items so that each one carries its own evidence and its own construction cost. That makes failure attribution and cost-per-valid-item measurable rather than assumed.

Positioning Against Prior Work

The native-source line motivates the work. MILU scraped about 1,500 exams into 79,000 questions but had to machine-translate about a quarter of its low-resource subject-language pairs with GPT-4o. SOMAJGYAAN hand-built 4,234 Bangla questions from national textbooks. ITALIC filtered 2.1 million Italian exam questions down to 10,000 using an LLM-judge panel. All three are closed-book and are capped by what exam boards happen to publish. We keep native sourcing but replace "find an existing exam question" with "generate from a native passage." The new step removes the supply cap and gives gold evidence.

Thunder Korean constrains our design. Its finding that translated LAMBADA breaks on Korean verb-final word order is why we never translate. Generation happens in the target language, from target-language text.

The human-in-the-loop line supplies our method. GIMMICK seeds GPT-4o from UNESCO heritage records and has 18 regional experts polish the output. It does not account for cost, its questions are largely in English, and it is purely parametric. The author-reviewer efficiency figure we aim to approach comes from Bae's dialogue work, which found review about 13 times more efficient than writing from scratch. That figure was measured on dialogue, not exam questions, so we treat it as a hypothesis to test rather than a given.

The overhead and retrieval line supplies our filtering tools. Promptagator discards generated queries that a retriever cannot map back to their source passage, called a round-trip consistency filter. The same paper reports that this filter was too strict on small specialized collections such as NFCorpus, which matters for small native corpora. UDAPDR uses an expensive model only for a few seed prompts and a cheap open model for bulk generation. MIRAGE-Bench replaces expensive judging with a cheap learned surrogate. It evaluates multilingual RAG only with oracle passages, which is exactly the limitation our end-to-end setting removes.

Research Program

We work in two languages chosen for contrast: Bangla, which is low-resource, and Italian, which is comparatively high-resource. For each we assemble a native source collection whose provenance overlaps existing benchmarks. For Bangla this is the national curriculum social-studies textbooks that SOMAJGYAAN drew on. For Italian it is publicly available civics, constitutional and administrative texts of the kind tested in ITALIC's exams. We split documents into passages, and these passages form both the generation seeds and the retrieval collection.

Generation follows the UDAPDR cost pattern. A frontier API model writes perhaps fifty seed questions per language. An open-weight model then generates in bulk, in-language: one multiple-choice question per passage, with the answer key, distractors and a pointer to the supporting sentence. Llama-3.1-70B is a candidate because ITALIC already used it as a judge for Italian.

Each draft then receives three cheap automatic signals:

  1. Retrievability. Can a retriever find the source passage from the question alone? This is Promptagator's round-trip test, used here as a soft score rather than a hard filter.
  2. Grounding. Does an entailment model judge the keyed answer supported by the passage?
  3. Key agreement. Do several open models answer the question correctly when given the passage?

A small calibrated classifier combines these signals into an estimated probability that an item is defective. Native reviewers see only items in the uncertain middle band. Confidently good items are accepted after spot-checks, and confidently bad items are discarded.

Every stage logs its cost: GPU-hours, API dollars and reviewer minutes. Every rejection is also logged, so rejection waste, the "multi-stage tax" the overhead literature complains about, is measured directly.

The output is a benchmark in which each item has a native question, a gold passage, a construction-cost record and a review history. We evaluate under three regimes:

  • Closed-book: no documents.
  • Oracle: the gold passage is supplied.
  • End-to-end: the model must use BM25, a multilingual dense retriever, or a dynamic retrieve-when-uncertain controller in the style of RaDIO.

The gap between closed-book and oracle measures missing knowledge. The gap between oracle and end-to-end measures retrieval failure. Errors that remain under oracle passages measure reading and language failure.

Baselines

The naive reference point is the existing native exam benchmarks scored closed-book. For Bangla we use SOMAJGYAAN, where Gemini 2.0 Flash reaches 0.73 average accuracy and DeepSeek-R1-Distill-14B reaches 0.56. For Italian we use ITALIC, where Claude 3.5 Sonnet reaches 92.35% zero-shot. For Indic context we note MILU, where GPT-4o reaches 74.74% overall. We found no reported result on Lacuna for retrieval-augmented accuracy on MILU, SOMAJGYAAN or ITALIC, so our open-book numbers on these sources will be first measurements, not comparisons.

The strongest construction competitor is manual native authoring from the same passages. Native experts write questions directly, matching how SOMAJGYAAN's annotators worked. This is the quality ceiling, and the test is whether triaged review comes close to it at much lower human time. Both arms use the same passage pool, the same reviewer pool and the same audit protocol.

The second construction competitor is a fully automatic pipeline: the same open-model drafts filtered only by the hard round-trip test, in the Promptagator style, with ITALIC's three-model judge panel added. It gets the same generation compute as our pipeline and zero human minutes.

The ablation that isolates our key ingredient is uniform review. Reviewers check a random subset of drafts, of the same total size the triage would send them, instead of the uncertainty-selected subset. This separates the value of where human attention goes from the value of human attention at all.

Triage "beats" the competitors if two conditions hold. First, its audited defect rate is statistically indistinguishable from manual authoring while using at most a fifth of the reviewer minutes per accepted item. Second, at equal reviewer budget, its defect rate is clearly lower than both uniform review and the fully automatic pipeline.

For the evaluation side, the strongest retrieval competitor is the dynamic controller adapted from RaDIO. On 2WikiMultihopQA with BM25, RaDIO reports exact match of 0.228 versus DRAGIN's 0.214. We expect its English-trained hallucination detector to transfer poorly, and we test that directly.

Why not just…?

Why not just attach retrieval to existing exam questions like MILU or SOMAJGYAAN? This is a real threat to half our contribution, so we run it. For SOMAJGYAAN we retrieve over the same textbooks the questions came from. Without gold evidence, though, a wrong answer under retrieval cannot be attributed to the retriever, the reader, or a question whose answer is not in the collection at all. Exam items also cannot grow beyond what boards publish, and public exam PDFs are already being absorbed into web crawls. If retrieval over existing items already yields clean attributions, our evidence-anchoring adds little, and we will say so. We expect the large share of unattributable errors to show the difference.

Why not just let a frontier model generate everything without human review? Unreviewed generation is known to produce accidental correct answers among the distractors. Qu et al. report a Faithful Score of 33.39 for GPT-3.5-turbo on RACE versus 25.64 for their distilled model, where lower is better. Our fully automatic arm measures the analogous defect rate in native languages directly. If that rate falls within the audit's noise of the triaged arm, then human review is unnecessary and our pipeline should drop it.

Why not just translate an English benchmark with GPT-4o? Thunder Korean shows translation can break a task's structure, and MILU's translated quarter is the compromise its own authors flag. We include a small translated control set and test whether it ranks models the same way native exam items do. We expect it not to.

Why not just rely on the round-trip filter instead of people? Promptagator reports that this filter discarded useful pairs on small specialized collections, and native corpora are small. It also cannot detect an item that is retrievable but has a culturally wrong key. The fully automatic arm and the triage ablation together quantify how much of the defect rate the filter alone misses.

Proposed Experiments and Decisive Checks

The first experiment is the construction comparison. We draw roughly 1,500 passages per language and run all four arms on them: manual authoring, fully automatic, uniform review and triaged review. Two independent native experts who did not take part in construction audit a random sample of about 300 accepted items per arm. They label each item for four defects:

  • wrong key,
  • ambiguous,
  • unanswerable from its passage,
  • unnatural or translation-like phrasing.

The audit also reports inter-auditor agreement, which addresses the concern that even native reviewers can reintroduce distortion. The headline outputs are defect rate and full cost per non-defective item, plotted as a cost-quality curve per language.

The second experiment is a validity check. We score about fifteen open and closed models on our items and on SOMAJGYAAN and ITALIC closed-book, then report rank correlation. A low correlation would mean passage-anchored items measure something other than what native exams measure, which would undercut the benchmark however cheap it is.

The third experiment is the gap decomposition. We report closed-book, oracle and end-to-end accuracy, with each retriever, for every model. Results are broken out by subject, with local law and history contrasted against general subjects, following MILU's split. Under oracle passages we also record whether generated explanations stay in the target language, which tracks the "language collapse" failure.

The proposal is decisive in three cases. If triaged review matches manual quality at well under a fifth of the human time, cheap native expansion is viable. If the closed-book cultural gap largely closes with oracle passages but reopens under real retrieval, the bottleneck for these languages is native retrieval, not model knowledge. If the gap persists even under oracle passages for low-resource Bangla but not for Italian, the problem is reading native text, and further retrieval work would be misdirected.