Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation

Tathagata Raha, Clement Christophe, Hamza Javed, Marco Af Pimentel, Shadab Khan, Praveenkumar Kanithi

Abstract

Traditional metrics like BLEU and BERTScore fail to capture semantic fidelity in generative text-to-text tasks. We adapt the Cross-Examination Framework (CEF) for a reference-free, multi-dimensional evaluation by treating the source and candidate as independent knowledge bases. CEF generates verifiable questions from each text and performs a cross-examination to derive three interpretable scores: Coverage, Conformity, and Consistency. Validated across translation, summarization and clinical note-generation, our framework identifies critical errors, such as content omissions and factual contradictions, missed by standard metrics. A key contribution is a systematic robustness analysis to select a stable judge model. Crucially, the strong correlation between our reference-free and with-reference modes validates CEF's reliability without gold references. Furthermore, human expert validation demonstrates that CEF mismatching questions align with meaning-altering semantic errors higher than with non-semantic errors, particularly excelling at identifying entity-based and relational distortions.

1. Introduction

Evaluating the semantic fidelity or "truthfulness" of text-to-text generation remains a significant challenge. Traditional surface-level metrics like BLEU and ROUGE often penalize valid paraphrasing while failing to detect critical factual swaps (e.g., changing "yes" to "no"). Even modern embedding-based metrics like BERTScore struggle with specific factual contradictions. Furthermore, most reliable metrics depend on expensive human-written "gold standard" references, which are often unavailable in specialized domains like medicine or for low-resource languages.

To address these gaps, we propose the Cross-Examination Framework (CEF), a task-agnostic, reference-free diagnostic tool. CEF provides interpretable scores that categorize failures into content omissions, hallucinations, or factual contradictions.

2. Methodology

The core principle of CEF is that if two texts contain the same information, they should yield identical answers to the same set of questions. The framework operates in three stages:

  1. Question Generation: An LLM judge generates $N$ closed-ended questions from both the source and the candidate. These are constrained to be "YES-only" questions, ensuring the answer is "YES" based strictly on the generating text.
  2. Cross-Examination: Questions from the source are posed to the candidate, and vice versa. The judge responds with "YES," "NO," or "IDK" (I Don't Know).
  3. Score Computation:
    • Coverage: Measures source information retention (based on "IDK" responses to source questions).
  4. Consistency: Measures hallucinations (based on "IDK" responses to candidate questions).
  5. Conformity: Measures factual contradictions (based on "NO" responses).

Through systematic robustness analysis, DeepSeek-V3 was selected as the optimal judge model for its stability across languages.

3. Reference-Free Validation

A critical finding of this work is the reliability of CEF in reference-free settings. By comparing scores generated using the source text against those generated using human references, we observed high correlations, particularly for Consistency and Coverage.

Correlation between reference-free and with-reference CEF scores for Conformity, Consistency, and Coverage.

As shown in Figure 1, Consistency ($r = 0.846$) and Coverage ($r = 0.845$) show strong alignment, validating that the source text alone can serve as a robust baseline for evaluating generated content fidelity.

4. Results and Discussion

CEF was validated across Machine Translation (NTREX), Abstractive Summarization (CNN/DailyMail), and Clinical Note Generation (ACI-Bench).

4.1 Cross-Language Performance

In translation tasks, CEF proved resilient to morphological and script complexity. While traditional n-gram metrics often degrade for languages like Arabic and Japanese, CEF maintained diagnostic depth.

Heatmap of CEF scores across various LLMs and language pairs, highlighting performance variations in Consistency and Coverage.

Figure 2 illustrates that while commercial models like Google and Azure Translate maintain high fidelity, smaller open-source models (e.g., qwen3-1.7b) show significant drops in Consistency and Coverage when translating into complex scripts like Arabic (en-ar) and Japanese (en-jp).

4.2 Error Alignment

Human validation benchmarks (WMT’25 and FRANK) confirm that CEF mismatching questions align with semantic errors at a rate 3.4 times higher than with non-semantic stylistic errors. Specifically, CEF achieved alignment rates of 92.1% for relational errors and 86.2% for entity errors in summarization—areas where word-overlap metrics typically fail.

5. Conclusion and Future Work

CEF offers a transparent, interpretable alternative to "black-box" scorers. While the computational cost is higher than simple LLM ranking (approx. 6–7x), the diagnostic granularity is invaluable for high-stakes domains. Future work will focus on distilling CEF into smaller, specialized models to reduce costs and exploring multi-hop reasoning to capture more complex factual relationships.

Summary updated

Abstract

Traditional metrics like BLEU and BERTScore fail to capture semantic fidelity in generative text-to-text tasks. We adapt the Cross-Examination Framework (CEF) for a reference-free, multi-dimensional evaluation by treating the source and candidate as independent knowledge bases. CEF generates verifiable questions from each text and performs a cross-examination to derive three interpretable scores: Coverage, Conformity, and Consistency. Validated across translation, summarization and clinical note-generation, our framework identifies critical errors, such as content omissions and factual contradictions, missed by standard metrics. A key contribution is a systematic robustness analysis to select a stable judge model. Crucially, the strong correlation between our reference-free and with-reference modes validates CEF's reliability without gold references. Furthermore, human expert validation demonstrates that CEF mismatching questions align with meaning-altering semantic errors higher than with non-semantic errors, particularly excelling at identifying entity-based and relational distortions.
BibTeX
@misc{raha2026cross,
  author = {Tathagata Raha and Clement Christophe and Hamza Javed and Hamza A Javed and Marco AF Pimentel and Shadab Khan and Praveenkumar Kanithi},
  title = {{Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation}},
  year = {2026},
  eprint = {2601.19350},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.19350},
}
Figure