Khmer PFGen-Bench: Building a Judge-Free Benchmark for Open-Ended Generation in a Low-Resource Language

Mengsay Loem, Kentaro Imajo, Ryota Ozaki, Masanori HiranoarXiv 2026 · arXiv

Evaluating large language models (LLMs) in low-resource languages is a notorious bottleneck. While "LLM-as-a-judge" (using a frontier model like GPT-4 to grade others) has become the industry standard for English, it often fails for languages like Khmer, where even the best models exhibit biases, hallucinations, or a lack of cultural nuance.

In their recent paper, researchers from Preferred Networks introduce Khmer PFGen-Bench, a "judge-free" benchmark designed specifically for open-ended generation in Khmer. By moving away from subjective LLM grading and toward a statistical, reference-based approach, they provide a more reliable roadmap for evaluating models in the "long tail" of global languages.

TL;DR

Khmer PFGen-Bench is the first judge-free benchmark for open-ended Khmer generation, using n-gram statistics and keyword rules instead of unreliable LLM judges. The study reveals that while post-trained models produce more valid references, they are less diverse; however, mixing different model families in the reference pool creates a stable, human-aligned ranking.

What Problem Does This Paper Address?

Evaluating open-ended text—like "Describe the uses of a Khmer krama (scarf)"—is difficult because there isn't just one right answer. In high-resource languages, we ask GPT-4 to grade responses. But for Khmer, the official language of Cambodia, LLM judges are often inconsistent or biased toward their own output styles.

Existing Khmer benchmarks are often limited to multiple-choice questions or simple exact-match tasks. Khmer PFGen-Bench tackles the challenge of multi-sentence, open-ended generation without needing a high-resource "judge" model at scoring time.

Methods

The researchers adapted the PFGen-Bench framework (originally built for Japanese). The core idea is the distributional hypothesis: a high-quality response should contain the same "information units" (n-grams and keywords) found in a large pool of other high-quality responses.

The scoring system consists of three pillars:

  1. Fluency (F): Measures how common the response's n-grams are across the reference pool.
  2. Truthfulness (T): Checks if the response's 3-grams are supported by the reference pool (note: this measures "consensus" rather than absolute factual truth).
  3. Helpfulness (H): Uses question-specific keyword rules (e.g., "must mention 'cotton' or 'silk'") defined by native speakers.
Figure 1: Khmer PFGen-Bench (left) and the conditions behind its scores that this paper tests (right). The PFGen- Bench scorer compares each response with separately generated references, without an LLM judge; numbers link each condition to the component it concerns.

A critical part of the method is building the Reference Pool. The authors used 9 different model families to generate over 18,000 reference answers. They then filtered these for length, format, and keyword presence to ensure quality.

Datasets

  • Questions: 50 original questions written directly in Khmer by a native speaker, covering 13 categories from the Cambodian school curriculum (e.g., Khmer language, history, geography, and health).
  • Reference Pool: 18,274 filtered reference answers generated by 9 post-trained model families (including Gemma, Qwen, Llama, and GPT variants).
  • Evaluation Set: 19 models were evaluated, including both the reference generators and "non-generators" (models used only for testing).
  • Human Validation: A native speaker rated 240 responses to provide a "gold standard" for validating the automated scorer.

Key Insights and Ideas

  • The Yield vs. Coverage Trade-off: Post-trained (instruction-tuned) models follow instructions better, leading to a higher "yield" of valid references. However, they are more repetitive. Base models produce more "garbage," but their successful outputs are more linguistically diverse.
  • Diversity via Mixing: To get the best reference pool, it is better to take a few samples from many different model families than many samples from a single model. This "family mixing" significantly broadens the n-gram coverage.
  • The Self-Enhancement Bias: Models tend to score higher if their own generated answers are included in the reference pool. To ensure fairness, the benchmark excludes a model's own references when calculating its score.
  • Stability in Low-Resource Settings: Even though the "valid reference yield" in Khmer is much lower (0–30%) than in Japanese (29–68%), the resulting model rankings remain remarkably stable once a threshold of ~40 references per question is met.
Figure 2: Held-out-family 3-gram coverage by reference count N (a) and family count k (b). The gray band spans the available single generators (only DeepSeek and Gemini at N = 160); panel (b) uses common questions across k at each N.

Key Results

  • Human Alignment: The automated benchmark scores showed a high correlation ($\rho = 0.94$) with native speaker rankings at the model level.
  • Ranking Stability: Model rankings were highly consistent ($\rho \ge 0.95$) across different subsets of references, proving the benchmark isn't overly sensitive to which specific models are in the reference pool.
  • Frontier Gap: The benchmark clearly distinguished between "frontier" models (like Gemini-4-argon and Claude-opus-5.5) and smaller or less capable models, which scored significantly lower on Helpfulness (H) and Truthfulness (T).
Figure 3: Rankings under alternative reference sets for the reference generators (top) and non-generators (bottom). Ranks are computed within each group; the leftmost condition is self-excluded scoring.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scale: The benchmark currently only covers 50 questions.
  • Human Bottleneck: Validation relied on a single native speaker; future work could involve multiple raters to measure inter-rater agreement.
  • Factual Accuracy: While the keyword rules help, the n-gram "Truthfulness" score measures consensus among models, which could still be wrong if all models share a common hallucination.
  • Scorer Comparison: The study did not directly compare the PFGen-Bench scorer against a GPT-4 judge on the same Khmer data, which would be a valuable head-to-head test of reliability.

By providing a framework that doesn't rely on expensive, potentially biased LLM judges, Khmer PFGen-Bench offers a scalable path for the community to improve AI for the world's many low-resource languages.

Summary updated

Abstract

Benchmarks for open-ended generation often rely on large language model (LLM) judges, whose reliability in low-resource languages is uncertain. We introduce Khmer PFGen-Bench, to our knowledge the first judge-free benchmark for multi-sentence answers to questions composed directly in Khmer, a low-resource language. Khmer PFGen-Bench adapts the scorer of PFGen-Bench, a benchmark for Japanese that scores responses with keyword rules and n-gram statistics from LLM-generated references. This scoring requires references that are both valid and varied, which LLMs may not produce in Khmer, so we examine how to build such references. We find that post-trained models can give more valid but less varied references than their base variants, and that mixing model families increases variety for the same number of references. Using references from post-trained models in nine families, the benchmark yields model rankings that are stable across reference sets and consistent with a native speaker's ratings. These findings offer guidance for building judge-free benchmarks in other low-resource languages.
BibTeX
@misc{loem2026khmer,
  author = {Mengsay Loem and Kentaro Imajo and Ryota Ozaki and Masanori Hirano},
  title = {{Khmer PFGen-Bench: Building a Judge-Free Benchmark for Open-Ended Generation in a Low-Resource Language}},
  year = {2026},
  eprint = {2611.09222},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09222},
}
Figure