Adaptive Two-Phase Finetuning LLMs for Japanese Legal Text Retrieval

Quang Hoang Trung, Nguyen Van Hoang Phuc, Le Trung Hoang, Quang Huu Hieu, Vo Nguyen Le DuyarXiv 2024 · DOI · arXiv

Introduction

Text Retrieval (TR) in the legal domain presents unique challenges, requiring models to interpret precise terminology and complex logical relationships between queries (e.g., contract clauses) and documents (e.g., statutory articles). In the context of the Japanese language, these challenges are amplified by linguistic complexity—such as flexible syntax and difficult word segmentation—and a scarcity of high-quality, native legal datasets. Standard dense retrieval models often struggle to distinguish between legally accurate documents and those that are merely topically similar.

To address these issues, this research introduces a novel two-phase finetuning pipeline for Large Language Models (LLMs) specifically optimized for Japanese legal text retrieval. By transitioning from broad "global contextualization" to "domain-specific deepening," the proposed method achieves state-of-the-art results on both native Japanese legal data and standard benchmarks.

Methodology

The core of the proposed system is a multi-stage retrieval pipeline built around a bi-encoder architecture using LLaMA-2-7B. To manage the computational demands of finetuning a 7B-parameter model, the authors employed QLoRA (Quantized Low-Rank Adaptation), 16-bit quantization, and Flash Attention, enabling training on consumer-grade hardware.

The two-phase retrieval pipeline: Global Contextualization followed by Domain-Specific Deepening.

The Two-Phase Finetuning Strategy

  1. Phase 1: Global Contextualization: The model is initially trained on a diverse mixture of documents, including human-labeled positives, negatives retrieved by BM25+, and "easy" in-batch negatives. This phase establishes a robust, broad understanding of the general domain.
  2. Phase 2: Domain-Specific Deepening: The model undergoes further refinement using hard negative mining. It focuses on documents that the Phase 1 model ranked highly but were actually incorrect. This curriculum-based approach forces the model to learn the subtle distinctions necessary for high-precision legal work.

Datasets

The study utilizes two primary datasets to validate the pipeline:

  • Native Japanese Legal Dataset: A new dataset comprising 743 Japanese law articles related to employment and 302 manually verified contract chunks. The authors used Gemini 1.5 Pro to generate synthetic contract sentences, which were then refined through a "human-in-the-loop" verification process to ensure legal accuracy.
  • MS MARCO: A standard large-scale English passage retrieval benchmark used to demonstrate the generalizability of the pipeline beyond the Japanese language.

Key Results and Insights

The proposed method consistently outperformed established baselines, including BM25+, BGE, and the original RepLLaMA.

  • Japanese Legal Retrieval: On the custom native dataset, the model achieved a Recall@3 of 64.36 and an MRR@10 of 82.54, surpassing the RepLLaMA baseline (60.26 and 81.58, respectively).
  • MS MARCO Performance: The pipeline reached a Recall@10 of 80.71, compared to RepLLaMA's 77.55, proving its effectiveness in English retrieval tasks.
  • The Power of Ensembles: Combining the two-phase model with BM25+ lexical scores pushed the MRR@10 to 86.81 on the Japanese dataset, highlighting the complementary nature of lexical and semantic search.

Conclusion

This research demonstrates that a curriculum of negatives—moving from easy to hard examples—effectively prevents overfitting while providing the precision needed for expert-level retrieval. Furthermore, the results underscore the importance of native (non-translated) datasets for legal reasoning. Future work will explore larger batch sizes, the integration of more advanced base models like LLaMA-3, and the addition of a cross-encoder reranking stage to further optimize top-k precision.

Summary updated

BibTeX
@misc{trung2024adaptive,
  author = {Quang Hoang Trung and Nguyen Van Hoang Phuc and Le Trung Hoang and Quang Huu Hieu and Vo Nguyen Le Duy},
  title = {{Adaptive Two-Phase Finetuning LLMs for Japanese Legal Text Retrieval}},
  year = {2024},
  eprint = {2412.13205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.13205},
}
Figure