TL;DR
This paper tackles the challenge of automatically summarizing lengthy Portuguese legal documents from the Supreme Court of Justice. The researchers explore and compare extractive (LexRank, SeburSum) and abstractive (MBART) summarization techniques, along with a hybrid approach, to find the most effective method for this domain. Their findings highlight the trade-offs between different methods and suggest a promising direction for future work combining the strengths of all three.
What Problem Does This Paper Address?
Legal documents are notoriously long and filled with specialized jargon, making it difficult and time-consuming for legal professionals and the general public to extract essential information. Currently, summaries are created manually by judges, a process that is both labor-intensive and prone to delays. This paper aims to automate this summarization process for Portuguese legal documents, specifically judgments from the Portuguese Supreme Court of Justice, to improve efficiency and accessibility. A key challenge is adapting existing summarization models, often trained on general text, to the unique characteristics of legal language and structure across different jurisdictions.
Methods
The researchers implemented and evaluated three main approaches:
- LexRank: An unsupervised, graph-based extractive method that ranks sentences based on their similarity to other sentences in the document. This served as a baseline due to its simplicity and effectiveness in identifying relevant sentences.
- SeburSum (simplified version): A summary-level extractive strategy that generates candidate summaries by combining top-ranked sentences and then selects the most representative one. This approach was investigated to see if it could improve upon sentence-level extraction.
- MBART: A transformer-based abstractive model capable of handling multilingual texts. It was used for its ability to generate fluent and coherent summaries by understanding sentence context and relationships.
- Hybrid Approach (LexRank + MBART): To address MBART's token limitation (1024 tokens) and ensure informative input, this approach first uses LexRank to rank sentences by relevance and then feeds the top-ranked sentences (up to the token limit) into MBART for abstractive summarization.
The performance of these methods was evaluated using ROUGE (ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum) and BERTScore (recall, precision, F1-score) metrics.
Datasets
The study utilized a custom dataset of 5000 legal documents from the Portuguese Supreme Court of Justice, covering "Cível" (Civil), "Criminal," and "Social" areas. For each document, a corresponding reference summary was provided for evaluation.
Two distinct datasets were created from these documents:
- Dataset 1: Comprised of the original 5000 legal documents and their summaries.
- Dataset 2: A subset of 3552 documents where only the "fundamentação de direito" (legal reasoning) and "relatório" (report) sections were used. This was done to investigate the relevance of specific sections for summarization.
Key Insights and Ideas
- Extractive methods (LexRank) struggle with exact sentence matching: While LexRank could identify relevant sentences, the low ROUGE scores indicated that the generated summaries didn't always perfectly match the reference summaries in terms of exact wording. However, BERTScore showed that LexRank could still capture the semantic essence of the original documents.
- Hybrid approach shows promise but faces challenges: The LexRank + MBART hybrid approach aimed to leverage the strengths of both extractive ranking and abstractive generation. While it produced results comparable to LexRank, some generated summaries were malformed (e.g., repeating words), suggesting that fine-tuning MBART for this specific task is crucial.
- Dataset 2 (specific sections) generally yields better results: Both LexRank and the hybrid approach showed slightly better performance on Dataset 2, which used only the "fundamentação de direito" and "relatório" sections. This suggests these sections might be more critical for effective summarization.
- SeburSum offers a different perspective on summary-level strategies: The simplified SeburSum implementation demonstrated that it's possible to achieve similar information content with fewer sentences. However, the overlap in chosen sentences with LexRank was moderate (33-50%), indicating distinct sentence selection strategies.
Key Results
- LexRank: Achieved ROUGE-1 scores of up to 0.359 for Dataset 1 and 0.401 for Dataset 2. BERTScore F1 scores ranged from 0.676 to 0.710 for Dataset 1 and 0.695 to 0.733 for Dataset 2.
- LexRank + MBART: Showed ROUGE-1 scores of 0.252 for Dataset 1 and 0.302 for Dataset 2. BERTScore F1 scores were 0.619 for Dataset 1 and 0.653 for Dataset 2.
- SeburSum (simplified): Achieved ROUGE-1 scores of 0.347 for Dataset 1 and 0.401 for Dataset 2. BERTScore F1 scores were 0.679 for Dataset 1 and 0.706 for Dataset 2.
Limitations and Future Work
Limitations:
- Malformed summaries: The MBART model, even in the hybrid approach, sometimes produced nonsensical outputs, which negatively impacted evaluation scores.
- LexRank's sentence selection: LexRank's reliance on sentence extraction meant it could miss key information if it wasn't part of the highest-ranked sentences.
- SeburSum's simplified version: The study used a simplified version of SeburSum, not its full contrastive learning framework, which might limit its potential.
Future Work:
The authors propose a three-phase model that combines the strengths of all explored approaches:
- Sentence Ranking: Use LexRank (potentially with improved sentence embeddings) to rank sentences by importance.
- Candidate Selection: Employ a version of SeburSum to select the best candidate summary from the ranked sentences.
- Abstractive Generation: Feed the selected candidate into the MBART model to generate a final, structured, and contextually rich summary.
Summary updated