TL;DR
This paper introduces a novel typology for classifying arguments within legal decisions, focusing on evidence types, credibility factors, and comparative weighing patterns. By analyzing U.S. veterans' disability claims, the authors aim to create a system that can be automatically applied to identify argument trends and assist legal professionals.
What Problem Does This Paper Address?
Argument mining, the process of automatically extracting arguments from text, faces significant challenges in complex domains like legal decision-making. Legal cases involve intricate processes, multiple participants, and a vast array of evidence types, each with its own rules for assessment. Existing argument mining approaches often struggle to capture the nuanced reasoning involved in weighing conflicting evidence and assessing credibility, especially when these processes are governed by specific legal rules. This paper addresses the need for an argument mining framework that is not only accurate and transferable but also interoperable with existing legal frameworks for evidence assessment.
Methods
The authors developed an empirical approach to derive a typology for arguments by analyzing adjudicatory decisions. Their methodology involved:
- Sentence Classification: Identifying and categorizing sentences into "finding sentences" (conclusions), "evidence sentences" (sources of information), and "reasoning sentences" (linking evidence to findings).
- Evidence Type Cataloging: Developing a typology of evidence types and sub-types commonly found in legal decisions, inspired by but distinct from formal legal rules like the Federal Rules of Evidence.
- Credibility Factor Identification: Analyzing reasoning sentences to identify factors that tribunals consider when assessing the credibility or trustworthiness of individual pieces of evidence.
- Pattern Discovery: Searching for recurring patterns or "soft rules" that tribunals use to comparatively weigh the probative value of conflicting evidence.
The goal is to create a typology that is flexible, transferable, and has a reasonable likelihood of automatic and accurate classification.
Datasets
The study utilized a dataset of 30 fact-finding decisions issued by the U.S. Board of Veterans' Appeals (BVA) between 2013 and 2016. These decisions were specifically chosen from adjudicated disability claims by veterans for service-related post-traumatic stress disorder (PTSD). The authors note that this dataset is publicly available on GitHub.
Key Insights and Ideas
- Beyond Formal Rules: The paper highlights that legal tribunals often rely on implicit "soft rules" and patterns for weighing evidence, which go beyond the strict, formal legal rules governing evidence admissibility.
- Credibility is Multifaceted: Assessing the credibility of evidence, whether lay testimony or medical records, involves a systematic consideration of various factors related to the source, basis, and content of the information.
- Comparative Weighing is Crucial: Tribunals don't just assess evidence in isolation; they actively compare conflicting pieces of evidence, often favoring certain types or sources over others based on established patterns.
- Interoperability is Key: For argument mining to be truly useful in legal contexts, the derived typologies and classifications must be designed to align with and be interpretable within existing legal rules and concepts.
Key Results
The study identified several key findings:
- Evidence Typology: A typology of evidence types and sub-types was developed, including Lay Testimony (Veteran, spouse/partner, other veteran, other non-veteran) and Medical Records (Pre-service, In-service, Post-service).
- Credibility Factors for Lay Testimony: Factors include the witness's demeanor, character, consistency, personal knowledge, awareness of other evidence, and the facial plausibility and consistency of their testimony.
- Credibility Factors for Medical Records: Factors include the qualifications and potential bias of the record's author, the basis of the record (e.g., personal observation), and the content's detail, consistency, and corroboration.
- Comparison Patterns: The study found patterns such as favoring competent medical opinions over lay testimony when diagnosing conditions, and generally preferring contemporaneous documentary evidence over later, potentially biased lay testimony. The "Benefit-of-the-Doubt Rule" in veterans' claims was also noted as a significant factor when evidence is equally balanced.
Limitations and Future Work
The authors acknowledge that their typology is derived from a specific domain (veterans' disability claims) and that further research is needed to ensure its broad applicability across other legal and non-legal domains. Future work includes:
- Developing a Typology for Entire Arguments: Building upon the evidence types, credibility factors, and comparison patterns to classify complete arguments.
- Probabilistic Models: Investigating the use of probabilistic models to predict credibility and trustworthiness of evidence, potentially extending beyond legal domains.
- Achieving Interoperability: Further developing techniques to ensure semantic types used for argument classification are directly compatible with legal rules governing evidence assessment.
- Quantitative Research: Generating a larger annotated dataset to drive future quantitative research and develop rule-based software for automatic classification.
Summary updated