Introduction
Automated summarization has seen significant progress, yet most benchmarks remain "news-heavy" and "English-centric." Datasets like CNN/DailyMail or XSum often suffer from "lead bias," where the most critical information is concentrated in the opening sentences. In contrast, the legal domain presents a much steeper challenge. Legal acts are exceptionally long, highly structured, and written in specialized "legalese."
To address these challenges, researchers have introduced EUR-Lex-Sum, a massive new dataset for legal document summarization covering all 24 official languages of the European Union. This resource features human-curated summaries that are both multilingual and paragraph-aligned, providing a rigorous benchmark for long-form, cross-lingual AI.
Dataset Construction
The authors constructed EUR-Lex-Sum by crawling the EUR-Lex platform, the official gateway to EU law. The pipeline involved several critical steps:
- Crawling & Filtering: Over 5.5 million web requests were processed to extract legal acts and their corresponding summaries. The team prioritized HTML versions to ensure high text quality and structural metadata, discarding lower-quality PDF scans.
- Alignment: A unique feature of EU law is content parity across all official languages. The authors leveraged this to create a "paragraph-aligned" dataset, ensuring that specific sections in a summary (e.g., in German) correspond exactly to the same sections in other languages (e.g., Greek or Finnish).
- Baselines: The researchers established benchmarks using two approaches: a Zero-shot Extractive method using a modified LexRank with multilingual sentence embeddings, and a Cross-lingual Abstractive pipeline using the Longformer Encoder-Decoder (LED) architecture to manage massive input lengths.
Dataset Characteristics and Scale
EUR-Lex-Sum represents a significant leap in scale and complexity. It contains up to 1,500 document/summary pairs per language, totaling nearly 32,000 pairs across 24 languages. A "Gold Standard" subset of 375 legal acts is available in every single language, facilitating consistent cross-lingual evaluation.
As shown in the distributions above, the documents in EUR-Lex-Sum are approximately 16 times longer than those in the standard CNN/DailyMail dataset. The mean compression ratio is about 16:1, though the high standard deviation indicates that for some extremely long acts, models must be exceptionally selective.
Key Insights: Abstractivity and Translation
A surprising finding was the high level of abstractivity in the dataset. Typically, as document length increases, summaries tend to become more extractive. However, EUR-Lex-Sum remains highly abstractive, meaning human authors significantly rephrased content rather than simply quoting the law.
The table above highlights the $n$-gram novelty, confirming that a large percentage of the summary content does not appear verbatim in the source text.
Furthermore, the researchers explored the power of translation. In English-Spanish experiments, translating a high-quality English summary into Spanish outperformed generating a summary directly from Spanish text using extractive methods. This suggests that "cross-lingual transfer" is a viable strategy for lower-resourced languages within the EU framework.
Results and Limitations
The modified LexRank baseline achieved ROUGE-1 scores between 24 and 43. Irish (ga) emerged as an outlier with a high score of 43.66, likely due to its smaller sample size and specific structural characteristics in current EU documentation.
Despite its strengths, the dataset presents several hurdles for future research:
- Extreme Length: Many acts exceed the context windows of current Transformer models (like BigBird or LED), necessitating advanced chunking or filtering techniques.
- Temporal Bias: The dataset is skewed toward post-1990 documents due to the availability of high-quality digital formats for more recent legal acts.
- Hierarchical Information: Future models could better exploit HTML structures (headers, sub-points) rather than treating legal text as a flat sequence.
EUR-Lex-Sum invites the NLP community to tackle the complexities of the legal domain, with code and data available on GitHub.
Summary updated