AILA: A Question Answering System in the Legal Domain

Weiyi Huang, Jiahao Jiang, Qiang Qu, Min YangIJCAI 2020 · OpenReview

Abstract

Question answering (QA) in the legal domain has gained increasing popularity for people to seek legal advice. However, existing QA systems struggle to comprehend the legal context and provide jurisdictionally relevant answers due to the lack of domain expertise. In this paper, we develop an Artificial Intelligence Law Assistant (AILA) for question answering in the domain of Chinese laws. AILA system automatically comprehends users' natural language queries with the help of the legal knowledge graph (KG) and provides the best matching answers for given queries. In addition, AILA provides visual cues to interpret the input queries and candidate answers based on the legal KG. Experimental results on a large-scale legal QA corpus show the effectiveness of AILA. To the best of our knowledge, AILA is the first Chinese legal QA system which integrates the domain knowledge from legal KG to comprehend the questions and answers for ranking QA pairs.

Introduction

General-purpose search engines and question-answering (QA) systems often fail in the legal domain due to a lack of precision and domain blindness. They frequently return lengthy, irrelevant documents rather than direct advice. Furthermore, legal language is highly specific; a general model may not recognize that a user asking about "copying an article" is seeking information on "copyright infringement disputes." Existing deep learning models for text matching struggle to comprehend jurisdictional context and nuanced relationships between legal entities without specialized domain expertise.

AILA (Artificial Intelligence Law Assistant) addresses these challenges by integrating a massive, manually curated legal knowledge graph with a "knowledge-enhanced interactive attention network." By grounding natural language queries in formal legal concepts, AILA outperforms standard deep learning models and provides visual explanations for its answers.

System Architecture and Methodology

AILA is built on a three-pronged modular approach designed to bridge the gap between informal user language and formal legal jargon.

The three-module architecture of the AILA system: Data Acquisition, Question Answering, and Interpretation.

Knowledge-Enhanced Interactive Attention Network

The core of AILA uses Bidirectional LSTMs (Bi-LSTMs) to create semantic representations of questions and answers, enriched with domain knowledge from a Legal Knowledge Graph (KG). This KG contains 42,414 legal concepts and 1,426 types of disputes, constructed in collaboration with legal professionals.

Co-attention Mechanism

Instead of analyzing questions and answers in isolation, AILA employs a co-attention mechanism to learn correlations between them. This allows the question to "supervise" the modeling of the answer across different representation subspaces, ensuring the context of the query directly influences the ranking of potential answers.

Interpretation Module

To enhance transparency, AILA includes a visualization layer that highlights matching legal entities and displays attention weights. This shows users which specific words in their query triggered the selected legal advice.

Experimental Evaluation

The system was evaluated using the LegalQA Corpus, a dataset of 139,468 QA pairs crawled from online Chinese legal forums. AILA was compared against several deep learning baselines, including CNN, Bi-LSTM, and BiMPM.

Comparison of AILA against baseline models using Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR).

As shown in the results, AILA achieved a Mean Average Precision (MAP) of 0.8301 and a Mean Reciprocal Rank (MRR) of 0.8866, significantly outperforming the best baseline, KABLSTM. These results demonstrate that integrating domain knowledge from a legal KG is highly effective for ranking QA pairs in the legal domain.

Conclusion and Future Work

AILA represents a significant step forward as the first Chinese legal QA system to integrate domain-specific KG knowledge for query comprehension. While the system shows superior performance and interpretability, future work will focus on reducing the manual labor required for KG construction and exploring generative models to synthesize novel legal reasoning from multiple sources.

Summary updated

Abstract

Question answering (QA) in the legal domain has gained increasing popularity for people to seek legal advice. However, existing QA systems struggle to comprehend the legal context and provide jurisdictionally relevant answers due to the lack of domain expertise. In this paper, we develop an Artificial Intelligence Law Assistant (AILA) for question answering in the domain of Chinese laws. AILA system automatically comprehends users' natural language queries with the help of the legal knowledge graph (KG) and provides the best matching answers for given queries. In addition, AILA provides visual cues to interpret the input queries and candidate answers based on the legal KG. Experimental results on a large-scale legal QA corpus show the effectiveness of AILA. To the best of our knowledge, AILA is the first Chinese legal QA system which integrates the domain knowledge from legal KG to comprehend the questions and answers for ranking QA pairs. AILA is available at http://bmilab.ticp.io:48478/.
BibTeX
@inproceedings{huang2020aila,
  author = {Weiyi Huang and Jiahao Jiang and Qiang Qu and Min Yang},
  title = {{AILA: A Question Answering System in the Legal Domain}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=9rzAnYZ4dWk},
}
Figure