TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial Domain

TL;DR

This paper introduces TermGPT, a novel multi-level contrastive fine-tuning framework designed to improve how Large Language Models (LLMs) understand specialized terminology in high-stakes domains like law and finance. By combining sentence-level and token-level contrastive learning, TermGPT significantly enhances the LLMs' ability to distinguish subtle semantic differences between domain-specific terms, leading to more accurate downstream task performance.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown remarkable capabilities in text generation, but they often struggle with a phenomenon called "isotropy" in their embedding spaces. This means that the representations of words and phrases can become too uniformly distributed, making it difficult for LLMs to discriminate between closely related, domain-specific terms. This is particularly problematic in fields like law and finance, where precise understanding of terminology is crucial for tasks like legal judgment prediction or financial risk assessment. Misinterpreting a single term can lead to significant real-world consequences, such as financial losses or regulatory penalties. The paper aims to bridge this gap by developing a method to improve LLMs' fine-grained understanding of specialized terminology.

Methods

TermGPT employs a two-pronged approach: data augmentation and multi-level contrastive learning.

1. Data Augmentation using Sentence Graphs: To create better training data, the authors construct a "sentence graph" that captures semantic and structural relationships between sentences. This graph is then used to generate semantically consistent yet discriminative positive and negative sample pairs. This helps address the ambiguity of supervision signals for terminology.
2. Multi-Level Contrastive Learning:
   • Sentence-Level Contrastive Learning: This component focuses on enhancing the LLM's global contextual understanding. It optimizes sentence embeddings to capture the broader meaning of terms within their context, preventing dilution of semantic information.
3. Token-Level Contrastive Learning: This component targets fine-grained terminology discrimination. It constructs positive and negative samples around key tokens, ensuring that important terms are not overshadowed by general contextual patterns. This is crucial for handling the sparsity of domain-specific terms.

The framework also incorporates supervised fine-tuning (SFT) as an initial step to align LLM outputs with instruction-response pairs.

Datasets

The paper utilizes two datasets for evaluation:

1. JecQA Dataset: A large-scale legal question-answering dataset.
2. Financial Regulations Dataset: A newly constructed dataset specifically for financial terminology adaptation. This dataset comprises 3,647 domain-specific terms extracted from 425 authoritative Chinese financial regulatory documents. This is noted as the first such dataset derived from official regulatory documents.

Key Insights and Ideas

• The "Isotropy Problem" is a Real Bottleneck: The paper clearly articulates how LLMs' inherent embedding space properties hinder their ability to grasp nuanced terminology, which is critical in high-stakes domains.
• Context vs. Specificity Trade-off: Terminology understanding requires balancing broad context with precise term meaning. TermGPT's multi-level approach directly addresses this by optimizing both sentence-level (context) and token-level (specificity) representations.
• Graph-Based Data Augmentation is Key: Constructing a sentence graph to guide data augmentation is an innovative way to generate more informative and discriminative training examples, especially for ambiguous or sparse terminology.
• Multi-Level is Better Than Single-Level: The experiments demonstrate that combining both sentence-level and token-level contrastive learning yields superior results compared to using either approach in isolation.

Key Results

• Superior Performance on QCA Tasks: TermGPT outperforms existing baselines by an average of 2.60% on terminology Question-Choice Answering (QCA) tasks. When compared to domain-pretrained LLMs (excluding large commercial ones), TermGPT shows a significant improvement of 6.14% on average for QA tasks.
• Effective Terminology Adaptation: The model achieves comparable performance to commercial LLMs on QA tasks, despite using smaller backbone models (8B parameters), highlighting the effectiveness of its targeted supervision.
• Ablation Studies Confirm Contributions: Removing either sentence-level or token-level contrastive learning leads to performance drops, indicating the necessity of both components. Removing contrastive learning entirely results in a larger drop (3.81%) than removing SFT (8.55%), suggesting SFT is foundational but contrastive learning is crucial for fine-grained discrimination.
• Robustness Across Domains: TermGPT demonstrates stable performance across various sub-domains within both legal and financial datasets, although some sub-domains with less training data show slightly lower scores.

Limitations and Future Work

• Sub-domain Performance Variance: While generally stable, performance can slightly dip in sub-domains with limited training data (e.g., Economic Law, Risk Management in QCA).
• Adversarial Robustness: The authors suggest that a crucial direction for future work is enhancing the robustness of terminology-aware LLMs against adversarial inputs, which could distort terminology interpretation in high-stakes scenarios. Future research could explore adversarial training or robust optimization techniques.