Effective Approach to Develop a Sentiment Annotator For Legal Domain in a Low Resource Setting

Gathika Ratnayaka, Nisansa de Silva, Amal Shehan Perera, Ramesh PathiranaarXiv 2020 · arXiv

Introduction

Sentiment analysis is a cornerstone of Natural Language Processing (NLP), yet its efficacy often diminishes when applied to specialized domains such as law. This performance degradation is primarily attributed to negative transfer, where models trained on general-purpose datasets—such as movie reviews—fail to account for the "emotional weight" shifts of specific terminology in a legal context. For instance, the word "charged" may denote excitement in a cinematic review but carries a strictly negative connotation in a courtroom setting.

Furthermore, the legal domain represents a "low-resource setting." The high cost of expert legal annotation makes the creation of large-scale labeled datasets prohibitively expensive. This research proposes a semi-automated framework to bridge this gap by identifying "sentiment-shifting" words, enabling targeted manual annotation with minimal human effort.

Methodology

The proposed approach adapts general-purpose sentiment models to the legal domain through a multi-stage pipeline:

  1. Word Categorization: Vocabulary is divided into Domain Generic (consistent meaning), Domain Specific (shifting meaning), and Under Represented (rare in general text but frequent in legal documents).
  2. Heuristic Identification: To detect sentiment shifts without exhaustive manual labeling, the researchers utilized Word2Vec embeddings trained on movie reviews (IMDB) and legal texts (SigmaLaw). A domainSimilarity score, based on cosine similarity between the most similar words in each domain, was used to flag words exhibiting divergent behavior.
  3. Targeted Annotation: Legal experts annotated only the flagged words and a small subset of sentences (925 words and 576 sentences), significantly reducing the manual workload.
  4. Model Adaptation:
  5. RNTN (Recursive Neural Tensor Network): Fine-tuned by replacing vectors of "deviated" words with vectors of words possessing the correct sentiment in the source domain.
  6. BERT: The training set was filtered to remove conflicting sentences, and the newly annotated legal data was integrated to create $BERT_l$.
Word distribution and identification precision across sentiment polarities.

Key Results

The effectiveness of the domain adaptation was evaluated by comparing the improved models ($RNTN_l$ and $BERT_l$) against baselines trained solely on source-domain data ($RNTN_m$ and $BERT_m$).

As shown in the performance data below, the results demonstrate a significant performance leap. The baseline $BERT_m$ achieved 57% accuracy on legal texts, while the adapted $BERT_l$ reached 67% accuracy—a 10% improvement achieved with fewer than 600 new annotations. Similarly, the $RNTN_l$ model saw an accuracy increase from 48% to 57%.

Performance comparison between baseline and improved models across Negative, Neutral, and Positive classes.

Beyond overall accuracy, the heuristic approach demonstrated high precision in identifying sentiment-shifting words. The system identified negative sentiment words with 80% precision, representing a substantial improvement over the baseline's internal assignments. The $RNTN_l$ model also showed much more consistent performance across all three classes (Positive, Neutral, Negative) compared to the movie-only models, which often struggled with the positive class in a legal context.

Discussion and Future Work

A critical insight from this study is that "Neutrality is a trap." Many words that carry sentiment in general usage become neutral in a legal context, and vice versa. Managing the "Neutral" class remains a challenge due to its sheer volume (nearly 5,000 words in the source set).

While the current study focuses on a 3-class sentiment problem (Positive, Neutral, Negative), future research could extend this heuristic approach to more granular scales (e.g., 5-class) or other low-resource domains like medicine or engineering, where specialized terminology similarly drifts from standard English usage.

Summary updated

Abstract

Analyzing the sentiments of legal opinions available in Legal Opinion Texts can facilitate several use cases such as legal judgement prediction, contradictory statements identification and party-based sentiment analysis. However, the task of developing a legal domain specific sentiment annotator is challenging due to resource constraints such as lack of domain specific labelled data and domain expertise. In this study, we propose novel techniques that can be used to develop a sentiment annotator for the legal domain while minimizing the need for manual annotations of data.
BibTeX
@misc{ratnayaka2020effective,
  author = {Gathika Ratnayaka and Nisansa de Silva and Amal Shehan Perera and Ramesh Pathirana},
  title = {{Effective Approach to Develop a Sentiment Annotator For Legal Domain in a Low Resource Setting}},
  year = {2020},
  eprint = {2011.00318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.00318},
}
Figure