Introduction
Evaluating document-level text simplification (DTS) is a significant challenge in natural language processing. Unlike sentence-level simplification, which often focuses on lexical substitution, DTS involves complex structural changes such as sentence reordering, splitting, fusion, and selective deletion. Current automatic metrics like SARI or LENS are primarily optimized for single-sentence pairs and struggle to capture these "many-to-many" relationships. Furthermore, many modern transformer-based metrics are limited by input length constraints, making them unsuitable for long documents.
The Challenge of Document-Level Evaluation
Standard metrics face two primary hurdles when applied to documents:
- Granularity and Alignment: Metrics often assume a linear 1:1 mapping between source and simplified sentences. In reality, a single complex sentence might be split into several simpler ones, or multiple sentences might be fused to improve flow.
- Technical Constraints: Models like RoBERTa, which underpin metrics like LENS, typically have a 512-token limit, preventing them from processing entire documents as a single input.
Methodology: A Divide-and-Conquer Framework
To address these issues, we propose a metric-agnostic aggregation framework that breaks long documents into manageable, semantically related "mini-units."
- Similarity Matrix Construction: We utilize a sentence-pair similarity model to compare every sentence in the source text against the simplified and reference versions.
- Graph-Based Alignment: We construct an undirected graph where sentences are nodes. Edges are created between sentences with a similarity score above 0.5. By identifying the "connected components" of this graph, we group related sentences across all versions of the text, effectively capturing many-to-many transformations.
- Metric Aggregation: Existing metrics (e.g., LENS, SARI, BERTScore) are applied to these aligned units. The resulting scores are averaged to produce a final document-level evaluation.
Experimental Results
We benchmarked our "Agg-metrics" against traditional sentence-level and document-level baselines across three datasets: COCHRANE (medical abstracts), D-WIKIPEDIA (encyclopedic entries), and ONESTOPQA (news articles).
As shown in the results table, Agg-LENS consistently outperforms other metrics. On the Cochrane dataset, it achieved a Kendall Tau correlation of 0.433, a dramatic improvement over the document-level baseline DSARI (-0.017).
Scaling with Document Length
A key advantage of the aggregation approach is its robustness as document length increases. While standard sentence-level metrics often see a decline in performance on longer texts, Agg-LENS maintains or even improves its correlation with human judgment as the number of tokens grows.
Discussion and Limitations
While aggregation improves sensitivity to structural changes and readability, it introduces a trade-off regarding "hallucinations"—cases where the model adds information not present in the source. Because scores are averaged across multiple units, a single unit containing a hallucination may be "diluted" by other correctly simplified units, making the metric less likely to penalize the error heavily.
Furthermore, our current framework is optimized for English and focuses on transformations like splitting and fusing. Future work will explore "elaborative simplification," where explanatory content is added, and aim to increase metric sensitivity to factual inconsistencies.
Conclusion
By adapting high-performing sentence-level metrics through a graph-based alignment strategy, we provide a more reliable way to evaluate document-level simplification. Agg-LENS, in particular, emerges as a robust tool for assessing readability and structural transformation in complex, long-form text.
Summary updated