SigmaLaw-ABSA: Dataset for Aspect-Based Sentiment Analysis in Legal Opinion Texts

Chanika Ruchini Mudalige, Dilini Karunarathna, Isanka Rajapaksha, Nisansa de Silva, Gathika Ratnayaka, Amal Shehan Perera, Ramesh Pathirana

TL;DR

This paper introduces SigmaLaw-ABSA, the first publicly available dataset for Aspect-Based Sentiment Analysis (ABSA) specifically tailored for legal opinion texts. By enabling fine-grained sentiment analysis of legal documents, this dataset aims to advance automated information extraction and analysis in the legal domain.

What Problem Does This Paper Address?

Extracting valuable legal information from opinion texts is crucial for legal professionals, but manual extraction is time-consuming. While sentiment analysis can identify opinions in text, it often struggles to attribute these sentiments to specific entities or parties within complex legal documents. Legal opinions frequently involve multiple parties, and understanding the sentiment expressed towards each individual party is essential for comprehensive analysis. Existing ABSA datasets are typically designed for domains like product reviews or movies and are not well-suited for the unique characteristics of legal texts, such as long sentences and complex subordinate structures. This gap necessitates a domain-specific dataset to facilitate research in legal ABSA.

Methods

The core contribution of this paper is the creation of the SigmaLaw-ABSA dataset. The authors collected legal opinion texts and manually annotated them for various ABSA sub-tasks relevant to the legal domain. These sub-tasks include:

  • Aspect Extraction (T1): Identifying all persons or organizations that belong to any legal party.
  • Aspect Term Polarity (T2): Assigning sentiment polarity (positive, negative, neutral) to the extracted aspects.
  • Aspect Category Extraction (T3): Categorizing the extracted aspects under specific legal parties (petitioner or defendant).
  • Aspect Category Polarity (T4): Determining the sentiment polarity for each aspect category.

The annotation process involved three human annotators, and inter-rater reliability was measured using Fleiss' kappa, achieving a moderate agreement level. The dataset also includes an "overall sentiment" feature for each sentence, independent of specific parties.

Datasets

  • SigmaLaw-ABSA: This is the newly introduced dataset, consisting of 2000 sentences (1007 full sentences and 993 sub-sentences) extracted from legal opinion texts. The data was collected from the SigmaLaw - Large Legal Text Corpus and Word Embeddings dataset, specifically focusing on United States Supreme Court cases. The dataset is annotated for aspect extraction, polarity detection, aspect category identification, and aspect category polarity.

Key Insights and Ideas

  • Domain Specificity is Crucial for ABSA: The paper highlights that general-purpose ABSA datasets are insufficient for the legal domain due to its unique linguistic features (long sentences, complex structures).
  • Party-Centric Sentiment Analysis is Key: In legal contexts, understanding sentiment towards specific parties (petitioners, defendants) is paramount, a nuance not adequately captured by standard sentiment analysis.
  • Multi-task Dataset Design: The SigmaLaw-ABSA dataset is designed to support multiple ABSA sub-tasks, offering flexibility for researchers to explore various aspects of legal text analysis.
  • Complexity of Legal Text: The authors acknowledge the inherent difficulty in handling long sentences and domain-specific phrases in legal documents, underscoring the importance of their annotated dataset.

Key Results

The paper presents a statistical overview of the SigmaLaw-ABSA dataset:

  • Total Sentences: 2000 (1007 full, 993 sub-sentences).
  • Overall Sentiment Distribution: 642 positive, 978 negative, 380 neutral sentences.
  • Party-Specific Sentiment Distribution:
    • Petitioner: 335 positive, 618 negative, 198 neutral.
  • Defendant: 285 positive, 244 negative, 106 neutral.

The authors also conducted baseline experiments using several deep learning models for aspect-based sentiment classification (Task T2). The Recurrent Attention on Memory (RAM) model achieved the best performance, with an accuracy of 0.663947 and an F1 score of 0.602201. This suggests that models capable of handling long-distance dependencies and complex attention mechanisms are well-suited for legal ABSA.

Limitations and Future Work

  • Limitations:
    • The inter-rater reliability for the annotation process, while moderate, indicates room for improvement in annotation consistency.
  • The dataset size of 2000 sentences, while substantial for a new legal ABSA dataset, might be considered moderate for training very complex deep learning models from scratch.
  • Future Work:
    • The authors express hope that the SigmaLaw-ABSA dataset will be instrumental in achieving the various ABSA sub-tasks within the legal domain.
  • They suggest that researchers can further explore the dataset to identify relationships between general sentiment analysis and Aspect-Based Sentiment Analysis.
  • The dataset is published publicly and hosted at OSF, encouraging further research and development in this area.

Summary updated

Abstract

Aspect-Based Sentiment Analysis (ABSA) has been prominent and ongoing research over many different domains, but it is not widely discussed in the legal domain. A number of publicly available datasets for a wide range of domains usually fulfill the needs of researchers to perform their studies in the field of ABSA. To the best of our knowledge, there is no publicly available dataset for the Aspect (Party) Based Sentiment Analysis for legal opinion texts. Therefore, creating a publicly available dataset for the research of ABSA for the legal domain can be considered as a task with significant importance. In this study, we introduce a manually annotated legal opinion text dataset (SigmaLaw-ABSA) intended towards facilitating researchers for ABSA tasks in the legal domain. SigmaLaw-ABSA consists of legal opinion texts in the English language which have been annotated by human judges. This study discusses the sub-tasks of ABSA relevant to the legal domain and how to use the dataset to perform them. This paper also describes the statistics of the dataset and as a baseline, we present some results on the performance of some existing deep learning based systems on the SigmaLaw-ABSA dataset.
BibTeX
@inproceedings{mudalige2020sigmalaw,
  author = {Chanika Ruchini Mudalige and Dilini Karunarathna and Isanka Rajapaksha and Nisansa de Silva and Gathika Ratnayaka and Amal Shehan Perera and Ramesh Pathirana},
  title = {{SigmaLaw-ABSA: Dataset for Aspect-Based Sentiment Analysis in Legal Opinion Texts}},
  booktitle = {Iciis},
  year = {2020},
  doi = {10.1109/iciis51140.2020.9342650},
}
Figure