Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learning

Xiaorong Wang, Ting Yang, Zhu Zhang, Shuo Wang, Zihan Zhou, Liner Yang, Zhiyuan Liu, Maosong SunarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) evolve to generate increasingly lengthy sequences, the challenge of evaluating their output quality has intensified. Traditional surface-level metrics like BLEU and ROUGE are insufficient for capturing semantic coherence or logical flow in long documents. Furthermore, direct LLM-as-a-judge approaches often suffer from the "lost in the middle" phenomenon, where models fail to synthesize information from the center of long contexts. To address these issues, we introduce Monocle, a hierarchical evaluation framework that combines local and global assessments with uncertainty-based active learning.

Methodology: The Monocle Framework

Monocle employs a "divide-and-conquer" strategy to manage the complexity of long-form text. The process is divided into two primary stages:

  1. Local Evaluation: The document is partitioned into manageable chunks. Local judges (LLMs) evaluate these chunks independently. To align model reasoning with human standards, we utilize Hybrid In-Context Learning (ICL). In this setup, the prompt includes human-annotated examples alongside model-generated explanations that justify the assigned scores.
  2. Global Evaluation: A global judge synthesizes the outputs from the local stage. Instead of re-reading the entire document, the global judge processes a concise summary of local scores and explanations, effectively bypassing context window limitations and preventing information loss.
The Monocle framework architecture, illustrating the transition from local chunk evaluation to global synthesis.

To optimize the human annotation process, Monocle incorporates an Uncertainty-Based Active Learning component. By measuring the variance (uncertainty) of model scores across multiple iterations, the system identifies the most challenging documents for human experts to annotate, maximizing the utility of each human label.

The ReliGen Benchmark

To validate Monocle, we developed ReliGen, a meta-evaluation benchmark focused on academic paper writing. ReliGen consists of 92 entries across diverse arXiv categories (e.g., Physics, CS, Math), featuring writing instructions, reference materials, and model-generated responses from state-of-the-art models like GPT-4o and Claude 3.5 Sonnet. Each entry is paired with high-quality human scores to serve as a ground truth for correlation analysis.

Experimental Results

Our experiments demonstrate that Monocle significantly outperforms existing evaluation methods in terms of alignment with human judgment.

Comparative performance of Monocle against traditional and LLM-based evaluation metrics on the ReliGen benchmark.

As shown in the results, Monocle achieved a Spearman’s rank correlation ($\rho$) of 0.568, which is a substantial improvement over the previous state-of-the-art LLM-based evaluator, HelloEval (0.275). Ablation studies confirmed that the hybrid local-global structure is essential; removing either component led to a marked decrease in correlation. Furthermore, the active learning strategy proved more efficient than random or perplexity-based selection, reaching higher accuracy with fewer human samples.

Conclusion

Monocle provides a robust and efficient solution for long-text evaluation by leveraging hierarchical judging and active learning. While currently training-free, future iterations could benefit from fine-tuning backbone models specifically for this task. Additionally, extending the framework to more subjective domains like creative writing remains a promising avenue for future research.

Summary updated

Abstract

Assessing the quality of long-form, model-generated text is challenging, even with advanced LLM-as-a-Judge methods, due to performance degradation as input length increases. To address this issue, we propose a divide-and-conquer approach, which breaks down the comprehensive evaluation task into a series of localized scoring tasks, followed by a final global assessment. This strategy allows for more granular and manageable evaluations, ensuring that each segment of the text is assessed in isolation for both coherence and quality, while also accounting for the overall structure and consistency of the entire piece. Moreover, we introduce a hybrid in-context learning approach that leverages human annotations to enhance the performance of both local and global evaluations. By incorporating human-generated feedback directly into the evaluation process, this method allows the model to better align with human judgment. Finally, we develop an uncertainty-based active learning algorithm that efficiently selects data samples for human annotation, thereby reducing annotation costs in practical scenarios. Experimental results show that the proposed evaluation framework outperforms several representative baselines, highlighting the effectiveness of our approach.
BibTeX
@misc{wang2025monocle,
  author = {Xiaorong Wang and Ting Yang and Zhu Zhang and Shuo Wang and Zihan Zhou and Liner Yang and Zhiyuan Liu and Maosong Sun},
  title = {{Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learning}},
  year = {2025},
  eprint = {2505.20195},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.20195},
}
Figure