In the world of legal tech, "close enough" isn't good enough. When a lawyer asks a retrieval system for the "penalties for late delivery" in a contract, they don't just want a list of clauses that mention "delivery." They need the specific penalty clause at the very top, followed by related but less direct clauses like liability caps or force majeure provisions.
Standard retrieval models often struggle with this "graded relevance." They are typically trained to distinguish between "relevant" and "irrelevant," treating all relevant documents as equally good. A new paper from researchers at DiliTrust introduces Cascading Circle Loss (CasCL), a framework designed to solve this exact problem by teaching models the subtle art of ranking.
TL;DR
CasCL is a novel contrastive learning framework that decomposes legal retrieval into two tasks: separating positives from negatives and preserving the relative order of "graded" positives. By using a cascade of self-paced circular boundaries, it achieves state-of-the-art performance on both binary retrieval and fine-grained ranking metrics without the typical trade-offs.
What Problem Does This Paper Address?
Legal retrieval is uniquely challenging because legal documents use highly standardized, repetitive language. This leads to high lexical overlap: a clause about "timely delivery" looks very similar to one about "late delivery penalties," even though they serve different legal functions.
Existing contrastive losses like InfoNCE are great at binary separation (Property 1 in the paper), but they ignore intra-positive ordering. They try to pull all relevant documents toward the query with equal force. This creates a "tension": should a tangentially related clause be pulled toward high similarity (as a positive) or left at moderate similarity (as its lower relevance suggests)? Forcing everything to the top clutters the results, leading to errors in downstream tasks like automated legal reasoning or information extraction.
Methods
The core innovation of CasCL is modular decomposition. Instead of one loss function trying to do everything, CasCL splits the work:
- InfoNCE Term (Scale Setting): This handles the "best" positive and all negatives, establishing the absolute boundary between relevant and irrelevant.
- Cascade Term (Order Preserving): This is a chain of self-paced circular boundaries that link adjacent items in a relevance ranking (e.g., $r=1.0 \succ r=0.8 \succ r=0.6 \dots \succ \text{hardest negative}$).
Key technical components include:
- Relevance Gain Weighting: Gradients are scaled by $2^{r_j}-1$, focusing the model's "attention" on correctly ordering the most relevant items—the ones that matter most for metrics like nDCG.
- Two-Regime Self-Pacing: A clever "emergency mode" prevents graded positives from getting "stranded" below negatives. If a positive item falls into the "negative zone," the loss applies a strong upward pull to rescue it.
- Proportional Margins: The "safe zone" or margin between two items in the ranking scales automatically based on the difference in their relevance scores.
Datasets
The authors evaluated CasCL on a proprietary multilingual legal clause retrieval dataset from DiliTrust.
- Scope: 150K training and 30K test examples across six languages (French, Portuguese, English, Italian, Spanish, and German).
- Labels: Each query is paired with one human-validated "best" positive. Intermediate "soft" positives were labeled using an ensemble of three LLMs (GPT-4, Qwen, Gemini) with a majority-vote strategy to ensure stability and reduce bias.
Key Insights and Ideas
- The Spring System Analogy: The cascade acts like a self-correcting spring system. If an item is too close to the query for its relevance level, it's pushed away; if it's too far, it's pulled closer. This positions every item at its "relevance proportional equilibrium."
- Bypassing the MAP/nDCG Trade-off: Usually, optimizing for ranking (nDCG) can hurt overall retrieval coverage (MAP/Recall). CasCL's two-regime mechanism (Figure 3) resolves this by keeping the two objectives independent yet complementary.
- LLM-as-a-Judge works at scale: By anchoring LLM scores to a human-validated "ground truth" positive, the researchers created a high-quality, graded supervision signal that would be prohibitively expensive to generate manually.
Key Results
CasCL consistently outperformed established baselines like Circle Loss, InfoNCE, and Margin MSE.
- Binary Metrics: It achieved a Recall@1 of 41.14 (a significant jump from InfoNCE's 37.95) and a MAP@10 of 55.63.
- Graded Metrics: It set a new ceiling for ranking quality, scoring 95.11 at NDCG@5 and 93.60 at NDCG@20.
- Ablation: Removing the InfoNCE term caused a "catastrophic collapse" in Recall@1 (dropping to 28.56), proving that the binary foundation is critical for the cascade to work.
Limitations and Future Work
While the results are impressive, the authors note several areas for further exploration:
- Domain Generality: The current evaluation is limited to legal data. Testing on standard benchmarks like MS MARCO or BEIR would strengthen the claim of domain agnosticism.
- Human-LLM Gap: While the LLM ensemble is robust, a small-scale human study is needed to quantify exactly how closely these automated labels align with expert legal judgment.
- Model Scaling: The framework was tested on a 0.6B parameter model (Qwen3-Embedding). It remains to be seen if the gains are even larger (or perhaps diminished) with massive 7B+ parameter encoders.
Summary updated