Evaluating large language models (LLMs) in low-resource languages is a notorious bottleneck. While "LLM-as-a-judge" (using a frontier model like GPT-4 to grade others) has become the industry standard for English, it often fails for languages like Khmer, where even the best models exhibit biases, hallucinations, or a lack of cultural nuance.
In their recent paper, researchers from Preferred Networks introduce Khmer PFGen-Bench, a "judge-free" benchmark designed specifically for open-ended generation in Khmer. By moving away from subjective LLM grading and toward a statistical, reference-based approach, they provide a more reliable roadmap for evaluating models in the "long tail" of global languages.
TL;DR
Khmer PFGen-Bench is the first judge-free benchmark for open-ended Khmer generation, using n-gram statistics and keyword rules instead of unreliable LLM judges. The study reveals that while post-trained models produce more valid references, they are less diverse; however, mixing different model families in the reference pool creates a stable, human-aligned ranking.
What Problem Does This Paper Address?
Evaluating open-ended text—like "Describe the uses of a Khmer krama (scarf)"—is difficult because there isn't just one right answer. In high-resource languages, we ask GPT-4 to grade responses. But for Khmer, the official language of Cambodia, LLM judges are often inconsistent or biased toward their own output styles.
Existing Khmer benchmarks are often limited to multiple-choice questions or simple exact-match tasks. Khmer PFGen-Bench tackles the challenge of multi-sentence, open-ended generation without needing a high-resource "judge" model at scoring time.
Methods
The researchers adapted the PFGen-Bench framework (originally built for Japanese). The core idea is the distributional hypothesis: a high-quality response should contain the same "information units" (n-grams and keywords) found in a large pool of other high-quality responses.
The scoring system consists of three pillars:
- Fluency (F): Measures how common the response's n-grams are across the reference pool.
- Truthfulness (T): Checks if the response's 3-grams are supported by the reference pool (note: this measures "consensus" rather than absolute factual truth).
- Helpfulness (H): Uses question-specific keyword rules (e.g., "must mention 'cotton' or 'silk'") defined by native speakers.
A critical part of the method is building the Reference Pool. The authors used 9 different model families to generate over 18,000 reference answers. They then filtered these for length, format, and keyword presence to ensure quality.
Datasets
- Questions: 50 original questions written directly in Khmer by a native speaker, covering 13 categories from the Cambodian school curriculum (e.g., Khmer language, history, geography, and health).
- Reference Pool: 18,274 filtered reference answers generated by 9 post-trained model families (including Gemma, Qwen, Llama, and GPT variants).
- Evaluation Set: 19 models were evaluated, including both the reference generators and "non-generators" (models used only for testing).
- Human Validation: A native speaker rated 240 responses to provide a "gold standard" for validating the automated scorer.
Key Insights and Ideas
- The Yield vs. Coverage Trade-off: Post-trained (instruction-tuned) models follow instructions better, leading to a higher "yield" of valid references. However, they are more repetitive. Base models produce more "garbage," but their successful outputs are more linguistically diverse.
- Diversity via Mixing: To get the best reference pool, it is better to take a few samples from many different model families than many samples from a single model. This "family mixing" significantly broadens the n-gram coverage.
- The Self-Enhancement Bias: Models tend to score higher if their own generated answers are included in the reference pool. To ensure fairness, the benchmark excludes a model's own references when calculating its score.
- Stability in Low-Resource Settings: Even though the "valid reference yield" in Khmer is much lower (0–30%) than in Japanese (29–68%), the resulting model rankings remain remarkably stable once a threshold of ~40 references per question is met.
Key Results
- Human Alignment: The automated benchmark scores showed a high correlation ($\rho = 0.94$) with native speaker rankings at the model level.
- Ranking Stability: Model rankings were highly consistent ($\rho \ge 0.95$) across different subsets of references, proving the benchmark isn't overly sensitive to which specific models are in the reference pool.
- Frontier Gap: The benchmark clearly distinguished between "frontier" models (like Gemini-4-argon and Claude-opus-5.5) and smaller or less capable models, which scored significantly lower on Helpfulness (H) and Truthfulness (T).
Limitations and Future Work
The authors acknowledge several limitations:
- Scale: The benchmark currently only covers 50 questions.
- Human Bottleneck: Validation relied on a single native speaker; future work could involve multiple raters to measure inter-rater agreement.
- Factual Accuracy: While the keyword rules help, the n-gram "Truthfulness" score measures consensus among models, which could still be wrong if all models share a common hallucination.
- Scorer Comparison: The study did not directly compare the PFGen-Bench scorer against a GPT-4 judge on the same Khmer data, which would be a valuable head-to-head test of reliability.
By providing a framework that doesn't rely on expensive, potentially biased LLM judges, Khmer PFGen-Bench offers a scalable path for the community to improve AI for the world's many low-resource languages.
Summary updated