Towards Cross-Domain Transferability of Text Generation Models for Legal Text

Vinayshekhar Bannihatti Kumar, Kasturi Bhattacharjee, Rashmi GangadharaiahNLLP @ EMNLP 2022 · DOI

TL;DR

This paper investigates how well text generation models, like T5 and BART, can summarize legal documents across different domains (e.g., bills, contracts, privacy policies). They found that current models struggle with cross-domain transfer, meaning a model trained on one type of legal text doesn't perform well on another. However, training models on a combination of domains significantly improves performance, and they also introduce a new dataset for privacy policy summarization.

What Problem Does This Paper Address?

Legal documents are notoriously complex and filled with jargon, making them difficult for non-experts to understand. Summarization is a key technique to improve accessibility. However, creating labeled datasets for every specific legal domain is a massive undertaking. This scarcity of data makes it challenging to train robust summarization models that can generalize across different types of legal texts. The paper tackles this challenge by exploring how well existing state-of-the-art text generation models can transfer their summarization capabilities from one legal domain to another, and whether combining multiple domains can mitigate this limitation.

Methods

The researchers leverage two prominent pre-trained sequence-to-sequence Transformer models: BART and T5. They employ two main experimental settings:

  1. Single-Domain Setting: Models are fine-tuned on a specific legal domain's dataset and then evaluated on test sets from the same domain (in-domain) and other legal domains (cross-domain). This directly measures transferability.
  2. Multi-Domain Setting: Models are fine-tuned on a combined dataset encompassing all four legal domains. This explores whether learning from diverse legal texts improves generalization.

They evaluate model performance using standard summarization metrics: ROUGE-L, METEOR, BERTScore, and BARTScore.

Datasets

The study utilizes four distinct legal text datasets:

  • BillSum: Summaries of US congressional bills.
  • JRC-Acquis (en): Summaries of EU legislation, treaties, and declarations, using titles as summaries.
  • Legal Contracts: Summaries of terms of service and licensing agreements, with section-level summaries.
  • Privacy Policies: A newly curated dataset for section summarization of privacy policies, created from a larger corpus of privacy policies. This is a novel contribution to the research community.

Key Insights and Ideas

  • Limited Cross-Domain Transfer: The most striking finding is the poor performance of models when transferring summarization capabilities between different legal domains. A model trained on one type of legal text struggles significantly when applied to another.
  • The Power of Multi-Domain Training: Combining data from multiple legal domains during training dramatically improves model performance across all domains, highlighting the benefit of learning from diverse legal language patterns.
  • Metric Discrepancies: The paper observes that different evaluation metrics can yield varying conclusions about model performance. While overlap-based metrics like ROUGE-L and METEOR show a significant drop in cross-domain scenarios, metrics like BERTScore and BARTScore are less sensitive to these drops, suggesting they might not fully capture the quality degradation in summarization. However, qualitative analysis indicates that ROUGE-L and METEOR might better reflect the actual quality differences.

Key Results

  • Low Cross-Domain Performance: For instance, a BART-base model trained on JRC-Acquis (en) achieved a ROUGE-L score of 0.769 on its own domain but dropped to 0.104 on the BillSum domain. Similar drops were observed for T5-base models.
  • Multi-Domain Improvement: The multi-domain models consistently outperformed their single-domain counterparts. For example, a multi-domain BART-base model achieved a ROUGE-L score of 0.794 on JRC-Acquis (en), compared to 0.769 for the best single-domain model.
  • Qualitative Evidence: Table 2 and Table 5 showcase generated summaries, demonstrating that multi-domain models produce summaries closer to the reference summaries, especially in cross-domain evaluations.

Limitations and Future Work

The paper acknowledges the limitations of current text generation models in achieving robust cross-domain transferability for legal text summarization. The authors suggest future work could explore:

  • Zero-shot and Few-shot Learning: Investigating how models can perform summarization with very little or no domain-specific training data.
  • Legal Text Generation Specifics: Developing techniques tailored specifically for the nuances of legal language generation.

Summary updated

BibTeX
@inproceedings{kumar2022towards,
  author = {Vinayshekhar Bannihatti Kumar and Kasturi Bhattacharjee and Rashmi Gangadharaiah},
  title = {{Towards Cross-Domain Transferability of Text Generation Models for Legal Text}},
  booktitle = {NLLP @ EMNLP},
  year = {2022},
  doi = {10.18653/v1/2022.nllp-1.9},
}
Figure