Compression Through a Context Lens: Evaluating Adaptation, Quantization, and Distillation in Arabic Legal LLMs

Md. Rafiul Biswas, George Mikros, Wajdi ZaghouaniarXiv 2026 · arXiv

Large Language Models (LLMs) are transforming legal tech, but their massive size makes local deployment—often a requirement for legal confidentiality—a major headache. While model compression techniques like quantization and distillation offer a way out, we don't fully understand how they impact domain-specific knowledge, especially in "lower-resource" but morphologically rich languages like Arabic.

A new paper, "Compression Through a Context Lens: Evaluating Adaptation, Quantization, and Distillation in Arabic Legal LLMs," tackles this by examining how compression affects models' "parametric knowledge" versus their "reading comprehension" in the Saudi legal domain.

TL;DR

This study reveals that standard "open-book" evaluations mask the performance degradation caused by model compression. By testing Arabic legal LLMs without providing supporting context, the authors found that 4-bit quantization is remarkably efficient at preserving legal knowledge, while the success of knowledge distillation depends heavily on the size of the student model.

What Problem Does This Paper Address?

Deploying LLMs in legal settings requires a delicate balance between performance and privacy. While cloud APIs are easy, local deployment is preferred for sensitive data. However, legal-grade models are often too large for standard hardware.

The authors identify a critical evaluation gap: most benchmarks provide a "supporting passage" (open-book style). In this setting, a model might perform well simply by extracting the answer from the text, hiding the fact that it has lost its underlying legal "intuition" or knowledge during compression. To truly understand if a compressed model is still a "legal expert," we need to test it "closed-book"—without any external context.

Methods

The researchers developed a three-stage experimental pipeline to evaluate three prominent models: FANAR-9B, JAIS-8B, and Qwen2.5-7B.

Figure 1: Overview of the proposed methodology.
  1. Stage 1: Domain Adaptation. They compared base models against those trained with Continued Pre-training (CPT), Supervised Fine-tuning (SFT), and a sequential combination (CPT→SFT) using parameter-efficient QLoRA.
  2. Stage 2: Post-Training Quantization. They compressed the best-adapted models into 8-bit (INT8) and 4-bit (NF4 and FP4) formats to see how much memory could be saved without losing accuracy.
  3. Stage 3: Knowledge Distillation. They transferred knowledge from the larger SFT teachers into smaller student models (ranging from 7B down to 0.5B parameters) using sequence-level hard-label distillation.

Crucially, every stage was evaluated using a Dual-Protocol:

  • With-context (Open-book): Question + Options + Legal Passage.
  • No-context (Closed-book): Question + Options only.

Datasets

The study utilized ArabLegalEval, a comprehensive benchmark based on Saudi Arabian legal sources.

  • MCQ Subset: 500 held-out multiple-choice questions used for the primary evaluation.
  • NajizQA: 200 examples for open-ended question answering to assess generative quality.

Key Insights and Ideas

  • The "Context Lens" Matters: Providing context creates a performance ceiling. In "open-book" tests, models often hit 98%+ accuracy, making it impossible to distinguish between a 4-bit quantized model and a full-precision one. The "no-context" protocol is the true stress test for model quality.
  • Quantization is a "Free Lunch": Moving to 4-bit (NF4) quantization reduced memory usage by up to 71% while only shifting no-context accuracy by -3.2 to +1.6 percentage points. This suggests that the legal knowledge is robustly distributed within the model weights.
  • Distillation Scaling is Harsh: Unlike quantization, distillation results were highly variable. While a 7B student could nearly match its teacher, accuracy plummeted as student size decreased (e.g., dropping from ~77% to ~36% when moving from a 7B to a 0.5B student).
  • Cross-Architecture Distillation Works: Interestingly, distilling knowledge from an Arabic-centric model (FANAR) into a general multilingual model (Qwen2.5) was effective, showing that specialized legal knowledge can be transferred across different model families.

Key Results

The paper provides a detailed breakdown of how adaptation and compression interact.

Table 1: Stage 1 results for three base models and four adaptation variants. MCQ accuracy is reported on the fixed 500-item set and QA ROUGE-L on 200 items; ∆denotes With-Ctx minus No-Ctx. Bold marks the best variant per model and protocol (ties both bolded). Raw counts and ROUGE-1 are given in Appendix Table 4
  • The Context Gap: For the base FANAR-9B model, removing the context caused a massive 27 percentage point drop in accuracy (from 97.8% to 70.8%).
  • Quantization Efficiency: 4-bit NF4 quantization consistently outperformed FP4. For JAIS-8B, NF4 actually improved no-context accuracy by 1.6 points while slashing memory from 15GB to 5GB.
  • Distillation Success: A Qwen2.5-7B student distilled from FANAR achieved 76.6% no-context accuracy, nearly matching its teacher's 77.2%, despite being a different architecture.

Limitations and Future Work

The authors acknowledge several constraints:

  • Scope: The study is limited to Saudi Arabian law and the Arabic language; results might differ for other jurisdictions or lower-resource languages.
  • Distillation Method: They only used "hard-label" distillation. Future work could explore "soft-label" distillation or quantization-aware training (QAT) for even better results.
  • Evaluation Size: The MCQ evaluation was performed on a subset of 500 items. While statistically significant, larger test sets would provide more granular insights into specific legal sub-domains.

For engineers and researchers, the takeaway is clear: if you are building domain-specific LLMs for deployment, stop relying solely on open-book benchmarks. To ensure your compressed model actually "knows" its stuff, you have to take away the cheat sheet.

Summary updated

Abstract

Model compression can facilitate the deployment of large language models in resource-constrained legal settings, but its effects on domain-specific performance may depend on the evaluation protocol. We investigate this issue using ArabLegalEval, a Saudi Arabic legal benchmark, under both with-context and no-context evaluation. The former provides supporting legal passages, while the latter assesses performance without external context. Removing the context produces accuracy gaps of up to 27 percentage points for the base model. We examine three stages: domain adaptation, quantization, and knowledge distillation across FANAR-9B, JAIS-8B, and Qwen2.5-7B. Four-bit quantization reduces model memory by 64-71%, with changes in no-context MCQ accuracy ranging from -3.2 to +1.6 percentage points. Distillation produces more variable results, with accuracy generally declining as student size decreases. A Qwen2.5-7B student distilled from FANAR achieves 76.6% no-context accuracy, compared with 77.2% for its teacher. Our findings suggest that no-context evaluation provides a useful complement to context-supported testing when assessing the effects of adaptation and compression on Arabic legal language models.
BibTeX
@misc{biswas2026compression,
  author = {Md. Rafiul Biswas and George Mikros and Wajdi Zaghouani},
  title = {{Compression Through a Context Lens: Evaluating Adaptation, Quantization, and Distillation in Arabic Legal LLMs}},
  year = {2026},
  eprint = {2611.07988},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07988},
}
Figure