Abstract
Traditional metrics like BLEU and BERTScore fail to capture semantic fidelity in generative text-to-text tasks. We adapt the Cross-Examination Framework (CEF) for a reference-free, multi-dimensional evaluation by treating the source and candidate as independent knowledge bases. CEF generates verifiable questions from each text and performs a cross-examination to derive three interpretable scores: Coverage, Conformity, and Consistency. Validated across translation, summarization and clinical note-generation, our framework identifies critical errors, such as content omissions and factual contradictions, missed by standard metrics. A key contribution is a systematic robustness analysis to select a stable judge model. Crucially, the strong correlation between our reference-free and with-reference modes validates CEF's reliability without gold references. Furthermore, human expert validation demonstrates that CEF mismatching questions align with meaning-altering semantic errors higher than with non-semantic errors, particularly excelling at identifying entity-based and relational distortions.
1. Introduction
Evaluating the semantic fidelity or "truthfulness" of text-to-text generation remains a significant challenge. Traditional surface-level metrics like BLEU and ROUGE often penalize valid paraphrasing while failing to detect critical factual swaps (e.g., changing "yes" to "no"). Even modern embedding-based metrics like BERTScore struggle with specific factual contradictions. Furthermore, most reliable metrics depend on expensive human-written "gold standard" references, which are often unavailable in specialized domains like medicine or for low-resource languages.
To address these gaps, we propose the Cross-Examination Framework (CEF), a task-agnostic, reference-free diagnostic tool. CEF provides interpretable scores that categorize failures into content omissions, hallucinations, or factual contradictions.
2. Methodology
The core principle of CEF is that if two texts contain the same information, they should yield identical answers to the same set of questions. The framework operates in three stages:
- Question Generation: An LLM judge generates $N$ closed-ended questions from both the source and the candidate. These are constrained to be "YES-only" questions, ensuring the answer is "YES" based strictly on the generating text.
- Cross-Examination: Questions from the source are posed to the candidate, and vice versa. The judge responds with "YES," "NO," or "IDK" (I Don't Know).
- Score Computation:
- Coverage: Measures source information retention (based on "IDK" responses to source questions).
- Consistency: Measures hallucinations (based on "IDK" responses to candidate questions).
- Conformity: Measures factual contradictions (based on "NO" responses).
Through systematic robustness analysis, DeepSeek-V3 was selected as the optimal judge model for its stability across languages.
3. Reference-Free Validation
A critical finding of this work is the reliability of CEF in reference-free settings. By comparing scores generated using the source text against those generated using human references, we observed high correlations, particularly for Consistency and Coverage.
As shown in Figure 1, Consistency ($r = 0.846$) and Coverage ($r = 0.845$) show strong alignment, validating that the source text alone can serve as a robust baseline for evaluating generated content fidelity.
4. Results and Discussion
CEF was validated across Machine Translation (NTREX), Abstractive Summarization (CNN/DailyMail), and Clinical Note Generation (ACI-Bench).
4.1 Cross-Language Performance
In translation tasks, CEF proved resilient to morphological and script complexity. While traditional n-gram metrics often degrade for languages like Arabic and Japanese, CEF maintained diagnostic depth.
Figure 2 illustrates that while commercial models like Google and Azure Translate maintain high fidelity, smaller open-source models (e.g., qwen3-1.7b) show significant drops in Consistency and Coverage when translating into complex scripts like Arabic (en-ar) and Japanese (en-jp).
4.2 Error Alignment
Human validation benchmarks (WMT’25 and FRANK) confirm that CEF mismatching questions align with semantic errors at a rate 3.4 times higher than with non-semantic stylistic errors. Specifically, CEF achieved alignment rates of 92.1% for relational errors and 86.2% for entity errors in summarization—areas where word-overlap metrics typically fail.
5. Conclusion and Future Work
CEF offers a transparent, interpretable alternative to "black-box" scorers. While the computational cost is higher than simple LLM ranking (approx. 6–7x), the diagnostic granularity is invaluable for high-stakes domains. Future work will focus on distilling CEF into smaller, specialized models to reduce costs and exploring multi-hop reasoning to capture more complex factual relationships.
Summary updated