J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain

Yiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L. A. Clarke, Weixing Shen

TL;DR

The J&H framework tests whether large language models (LLMs) make legal judgments through genuine deductive reasoning or merely rely on surface-level correlations. By injecting subtle, knowledge-level perturbations into the major premises, factual minor premises, and conclusions of legal cases, the study shows that leading general and domain-specific LLMs are highly brittle. Strikingly, popular prompt-level fixes like Chain-of-Thought (COT) or few-shot examples fail to restore robustness, revealing that prompt engineering cannot substitute for sound domain reasoning.

What Problem Does This Paper Address?

Deploying LLMs as autonomous agents or assistants in high-stakes, knowledge-intensive domains (such as law or medicine) assumes that models reason using verified domain logic. In criminal law, a human judge uses classical deductive syllogism:

  1. Major Premise: Applicable statutory law.
  2. Minor Premise: Legally relevant case facts.
  3. Conclusion: Guilt or specific charge.

Human experts easily ignore irrelevant distractions, such as stylistic prose, minor factual typos, or biased lay opinions. If an LLM acts as an "LLM-as-judge," does it maintain logical causal inference, or is it swayed by superficial text cues? Prior robustness benchmarks mostly focused on character- or token-level noise; this work investigates whether LLMs can withstand attacks explicitly crafted to corrupt domain logic.

Methods

The authors introduced J&H (named after Jekyll & Hyde, representing Justice & Hellion), an adversarial evaluation framework targeting each stage of legal syllogistic deduction:

  • Major Premise Attacks: Misleading legal statutes are injected into the prompt context via simulated RAG (retrieval-augmented generation), or confusing similar crime definitions are introduced directly into the instructions to test if the model can still identify the correct applicable statute.
  • Minor Premise Attacks: Case facts are modified without changing the underlying legal reality. This includes:
  • Word Attacks: Replacing legal terms or ordinary words with legal/common synonyms.
  • Element Attacks: Appending adversarial factual or statutory elements borrowed from legally similar crimes.
  • Narration Attacks: Introducing environmental narrative color (e.g., "It was a stormy day" or "It's a dark night, a perfect time for murder").
  • Conclusion Level Attacks: Adding logically irrelevant assertions directly before asking for the verdict, including:
  • Previous Behavior Attacks: Mentioning unrelated past offenses by the defendant (which Chinese criminal law excludes from current guilt determination).
  • Expert Opinion Attacks: Inserting subjective opinions attributing the act to a similar crime from characters spanning laypersons and pupils to lawyers and judges.

The authors also tested the spatial sensitivity of attacks (inserting noise at the beginning, middle, or end of prompts) and evaluated three standard defenses: RAG, Chain-of-Thought (COT), and Few-shot prompting.

Datasets

The authors evaluated the framework on two Chinese criminal law datasets:

  • CAIL2018: 15,806 case documents spanning 184 criminal charges.
  • LEVEN: 3,323 legal event detection documents covering 61 criminal charges.

Cases were formatted into four-option multiple-choice questions (one ground-truth charge and three confusingly similar counter-options annotated by ten law school graduate students).

Key Insights and Ideas

  • LLMs crumble most at the conclusion stage: Irrelevant outside opinions and past criminal history caused the largest performance drops. When an injected sentence claimed an authority figure (lawyer or judge) believed the defendant committed an alternate crime, models frequently abandoned the correct verdict.
  • Authority bias is real for LLMs: Injected opinions attributed to judges or lawyers degraded accuracy significantly more than opinions attributed to laypersons or children.
  • Chain-of-Thought can backfire: Explicitly instructing models to reason step-by-step through the four elements of criminal law (subject, subjective aspect, objective aspect, object) frequently degraded robustness. LLMs hallucinated flawed legal logic chains that validated the injected distractor.
  • The "Lost in the Middle" effect applies to attacks: Adversarial statements injected into the beginning or end of prompts caused the largest drop in accuracy; adversarial noise hidden in the middle had a substantially reduced effect.
  • Domain-specific pre-training helps, but not enough: The legal LLM (Farui) outperformed general models (e.g., GPT-3.5-turbo, LLaMA-3), but remained susceptible to knowledge injection attacks.

Key Results

Experiments evaluated five models: GPT-3.5-turbo, Baichuan2-7B-chat, ChatGLM3-6B, LLaMA-3 (Chinese fine-tuned), and Farui (Alibaba's legal LLM), measured by Performance Drop Ratio ($\text{PDR} = 1 - \frac{\text{AttackAcc}}{\text{OriginalAcc}}$):

  • General Vulnerability: Adversarial attacks induced massive degradation across all models, with PDRs frequently exceeding 30% to 40%.
  • Major Premise Attacks: Supplying provisions for a similar, incorrect crime via RAG caused LLaMA-3’s accuracy to drop from 67.9% to 47.1% (a 30.63% PDR).
  • Minor Premise Attacks: In LEVEN, inserting statutory elements from similar crimes dropped Baichuan2's accuracy from 77.7% to 53.4% (31.27% PDR) and LLaMA-3's from 67.9% to 43.0% (36.67% PDR).
  • Conclusion Attacks: On LEVEN, attributing an incorrect charge to a "judge" dropped Farui's accuracy from 84.9% to 55.5% (34.63% PDR) and LLaMA-3's to 37.9% (44.18% PDR).
  • Defense Limitations: In element attack scenarios, COT increased PDR for Baichuan2 (provisional attack PDR worsened from 31.27% to 33.42%) and Farui (12.13% to 15.14%).

Limitations and Future Work

  • Language and Legal Jurisdiction: The experiments focus on statutory Chinese criminal law, leaving common-law and multi-lingual jurisdictions unexamined.
  • Prompt Mitigations Are Insufficient: The paper shows that prompting tricks cannot patch foundational reasoning deficits.
  • Future Work: The authors advocate moving beyond prompt-level defenses by directly integrating domain-specific deductive logic chains into the model pre-training and supervised fine-tuning phases. Additionally, they suggest applying the J&H syllogistic attack framework to other critical domains, such as medical diagnosis.

Summary updated

Abstract

As the scale and capabilities of Large Language Models (LLMs) increase, their applications in knowledge-intensive fields such as legal domain have garnered widespread attention. However, it remains doubtful whether these LLMs make judgments based on domain knowledge for reasoning. If LLMs base their judgments solely on specific words or patterns, rather than on the underlying logic of the language, the “LLM-as-judges” paradigm poses substantial risks in the real-world applications. To address this question, we propose a method of legal knowledge injection attacks for robustness testing, thereby inferring whether LLMs have learned legal knowledge and reasoning logic. In this paper, we propose J&H: an evaluation framework for detecting the robustness of LLMs under knowledge injection attacks in the legal domain. The aim of the framework is to explore whether LLMs perform deductive reasoning when accomplishing legal tasks. To further this aim, we have attacked each part of the reasoning logic underlying these tasks (major premise, minor premise, and conclusion generation). We have collected mistakes that legal experts might make in judicial decisions in the real world, such as typos, legal synonyms, inaccurate external legal statutes retrieval. However, in real legal practice, legal experts tend to overlook these mistakes and make judgments based on logic. However, when faced with these errors, LLMs are likely to be misled by typographical errors and may not utilize logic in their judgments. We conducted knowledge injection attacks on existing general and domain-specific LLMs. Current LLMs are not robust against the attacks employed in our experiments. In addition we propose and compare several methods to enhance the knowledge robustness of LLMs. All code can be found at the link.
BibTeX
@inproceedings{hu2025j,
  author = {Yiran Hu and Huanghai Liu and Qingjing Chen and Ning Zheng and Chong Wang and Yun Liu and Charles L. A. Clarke and Weixing Shen},
  title = {{J\&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=TbvgYAIfuT},
}

Loading related content...

Figure