TL;DR
This paper tackles the challenge of quantifying gender inequality in Indian divorce court proceedings using NLP. By developing novel methods to analyze a large corpus of court documents and carefully accounting for model biases, they reveal significant gender disparities, with women disproportionately experiencing domestic violence and dowry-related abuse.
What Problem Does This Paper Address?
While public court records offer a rich source for social science research, existing Natural Language Processing (NLP) tools can inadvertently introduce or amplify biases, leading to inaccurate conclusions about societal issues like gender inequality. Specifically, this paper addresses the problem of accurately measuring gender inequality in the context of Indian divorce proceedings, a topic often shrouded in social stigma and thus underrepresented in traditional surveys. The authors highlight that standard NLP methods might fail to capture the nuances of legal language and societal realities, potentially misrepresenting the extent and nature of gender bias.
Methods
The researchers employ two primary NLP methods:
- Word Embedding Association Test (WEAT): They adapt the traditional WEAT to analyze legal texts. A key innovation here is the disambiguation of "perpetrator" and "victim" roles within sentences. By using part-of-speech tagging and dependency parsing, they can differentiate between a husband torturing a wife and vice-versa, a crucial step for accurate bias measurement in this domain. They also introduce a modified dataset,
Dreplaced, where gendered identifiers are replaced with specific tokens to represent "male perpetrator," "female perpetrator," "male victim," and "female victim." - Text Entailment Framework: This method leverages Natural Language Inference (NLI) models. The core idea is to measure how often a given hypothesis (e.g., "A woman tortures a man") is entailed by sentences in the corpus. They introduce an "entailment gap" to quantify gender inequality, comparing the entailment ratios for hypotheses where the gender roles are swapped. To address model bias, they propose a "bias mitigation via inconsistency sampling" technique. This involves creating counterfactual examples by flipping gendered words in sentences and then training an NLI model to be consistent across these flipped pairs.
Datasets
- Divorce Court Proceedings Corpus (
Ddivorce): A newly created dataset comprising 17,306 divorce court proceedings scraped fromindiankanoon.org, covering January 1, 2012, to December 31, 2021. This corpus was filtered and processed to ensure relevance to divorce cases, resulting in approximately 30 million tokens. Dreplaced: A modified version ofDdivorcewhere gender identifiers are replaced with specific tokens to facilitate perpetrator/victim role disambiguation.
Key Insights and Ideas
- Subtlety of Legal Language: Legal texts present unique challenges for NLP due to their domain-specific nature and complex sentence structures, requiring careful adaptation of standard methods.
- Perpetrator vs. Victim Disambiguation is Crucial: Standard WEAT without distinguishing between the roles of perpetrator and victim can lead to near-perfect gender equality scores, masking actual societal biases.
- Women as Disproportionate Victims: The analysis reveals a striking gender inequality, with women frequently identified as victims of domestic violence, dowry demands, and torture.
- LLM Biases are Pervasive: Even sophisticated LLMs exhibit implicit gender biases, particularly in agent-theme relationships, where men are more likely to be perceived as agents with power.
- NLI as a Powerful Tool for Social Inference: Text entailment offers a fine-grained lens to analyze gender bias, and the proposed bias mitigation strategy can significantly improve the reliability of these analyses.
Key Results
- WEAT Analysis: When distinguishing between perpetrators and victims, the WEAT scores show that women are more frequently associated with unpleasant verbs (as victims), while men are more frequently associated as perpetrators. The baseline WEAT score (without disambiguation) was close to zero, indicating near-perfect equality, which the authors argue is misleading.
- Text Entailment Analysis: For most unpleasant verbs (e.g., abuse, assault, beat, burn, cheat, rape, slap, threaten, torture), men are identified as perpetrators more often than women. Verbs indicating physical harm (rape, beat) showed particularly high entailment gaps.
- LLM Bias Audit: LLMs show significant implicit bias in agent-theme relationships, with men more likely to be perceived as agents. The bias mitigation strategy using inconsistency sampling successfully reduced the bias score of an NLI model from 0.269 to 0.103 after three iterations.
Limitations and Future Work
- Sentence-Level Processing: The current analysis operates at the sentence level. Future work could explore document-level analysis to capture how contradicting claims within a single proceeding are resolved.
- Archival Limitations: The dataset's coverage might be limited by the archival extent of
indiankanoon.organd the sparse representation of certain regions. - Economic Independence: While some court proceedings mention occupations, a deeper exploration of economic independence and its link to divorce warrants further investigation.
- Out-of-Court Settlements: The study focuses on cases that reach court. Many unhappy marriages may not result in divorce, and the experiences of women in out-of-court settlements or unhappy marriages remain uncaptured.
- Binary Gender Assumption: The study acknowledges the risk of oversimplifying gender and notes the need for extensions to account for non-binary gender identities and same-sex marriages, which are not legally recognized in India.
Summary updated