Abstract
Legal relations form a highly consequential analytical framework of civil law system, serving as a crucial foundation for resolving disputes and realizing values of the rule of law in judicial practice. However, legal relations in Chinese civil cases remain underexplored in the field of legal artificial intelligence (legal AI), largely due to the absence of comprehensive schemas. In this work, we firstly introduce a comprehensive schema, which contains a hierarchical taxonomy and definitions of arguments, for AI systems to capture legal relations in Chinese civil cases. Based on this schema, we then formulate legal relation extraction task and present LexRel, an expert-annotated benchmark for legal relation extraction in Chinese civil law. We use LexRel to evaluate state-of-the-art large language models (LLMs) on legal relation extractions, showing that current LLMs exhibit significant limitations in accurately identifying civil legal relations. Furthermore, we demonstrate that incorporating legal relations information leads to consistent performance gains on other downstream legal AI tasks.
Introduction
In the legal domain, understanding a case requires more than identifying entities; it necessitates grasping legal relations—the specific rights and obligations between parties as defined by law (e.g., a creditor-debtor relationship). Historically, Legal AI has focused on basic factual entities or general social relations, largely ignoring formal legal relations due to the lack of standardized, fine-grained schemas. Without the ability to identify these underlying structures, AI systems struggle with deep legal reasoning, essentially reading facts without understanding the legal framework binding them together.
Methodology and Data Construction
To bridge this gap, the authors developed a structured schema and a novel extraction pipeline. The schema covers 9 major legal domains and 265 specific relation types, with precise definitions for three core arguments: Subject (the right/obligation holder), Object (the target, such as property), and Content (the substantive nature of the duty).
Legal Relation Extraction is formulated as a two-step process: identifying the correct legal relation type and then extracting the specific arguments. To build the LexRel benchmark—which contains 1,140 real-world Chinese civil cases and 1,863 annotated relations—the team utilized a "model-in-the-loop" approach. DeepSeek-V3 generated draft annotations that were rigorously corrected by six human legal experts.
Key Insights and Results
The Long-Tail of Law
Analysis of 26.6 million Chinese civil court judgments reveals that legal relations in the real world are highly skewed. The authors found that just 25 case types account for 80% of all civil judgments. The LexRel dataset naturally mirrors this long-tail distribution, providing a realistic but challenging environment for AI evaluation.
Benchmarking LLM Performance
Evaluation of top-tier LLMs shows that while frontier reasoning models (like o3-mini) can identify relation types, extracting specific arguments from unstructured text remains difficult. OpenAI's o3-mini led zero-shot performance with a micro-F1 of 0.762 on Type Extraction but only 0.382 on Argument Extraction.
However, Supervised Fine-Tuning (SFT) on synthetic data yielded massive improvements for smaller models. For example, InternLM3-8B's argument extraction micro-F1 skyrocketed from 0.048 to 0.323 after SFT.
Legal Relations as Structural Anchors
Explicitly modeling legal relations significantly boosts performance on downstream tasks. When models are provided with legal relations alongside case facts, their performance on tasks like statute prediction and document proofreading consistently improves. These relations act as a reasoning scaffold, helping models ground their outputs in actual legal doctrine rather than simple semantic matching.
Conclusion and Limitations
LexRel establishes a vital foundation for Chinese legal AI. While current LLMs exhibit limitations in zero-shot settings, fine-tuning and explicit relation modeling offer a path toward robust legal reasoning. Future work is required to adapt this schema to other jurisdictions, as the current framework is strictly tailored to the Chinese civil law system.
Summary updated