J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain

Yiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L. A. Clarke, Weixing Shen

Abstract

As Large Language Models (LLMs) are increasingly deployed in knowledge-intensive fields like law, their reasoning capabilities come under scrutiny. It remains unclear whether LLMs make judgments based on domain knowledge and logic or merely on surface-level patterns. This paper introduces J&H, an evaluation framework designed to probe the robustness of LLMs through legal knowledge-injection attacks. By targeting the components of legal syllogism—major premise, minor premise, and conclusion—we demonstrate that current LLMs, including domain-specific ones, are highly vulnerable to misleading information. Our findings suggest that LLMs often prioritize word patterns over deductive logic, posing significant risks for the "LLM-as-judge" paradigm.

Introduction: The "LLM-as-Judge" Dilemma

In the legal domain, judicial reasoning typically follows a strict syllogism: a major premise (the law), a minor premise (the facts of the case), and a conclusion (the judgment). While LLMs have demonstrated proficiency in legal examinations, it is uncertain if they are truly performing deductive reasoning or simply engaging in sophisticated pattern matching. If an LLM's judgment hinges on specific keywords rather than legal logic, it becomes susceptible to "knowledge-injection attacks"—where irrelevant or slightly incorrect legal information is introduced via prompts or retrieval systems (RAG). The J&H framework aims to determine whether LLMs act as "Justice" (logic-driven) or "Hellions" (correlation-driven).

The J&H Framework

The J&H framework systematically attacks each stage of the legal syllogism to evaluate model robustness.

Overview of the J&H framework for evaluating LLM robustness under knowledge-injection attacks.

1. Major Premise Attacks

These attacks target the legal statutes. Models are provided with incorrect statutes via RAG or mentions of "similar crimes" to see if they can still identify the correct law based on the facts.

2. Minor Premise Attacks

These target the factual basis of the case through three methods:

  • Word Attack: Replacing legal terms with synonyms.
  • Element Attack: Inserting adversarial factual elements from similar but distinct crimes.
  • Narration Attack: Adding "flavor text" (e.g., "it was a fine day" vs. "it was a dark night") to test if semantic "vibes" alter the verdict.
Examples of Minor Premise attacks, including word, element, and narration attacks on a legal case fact.

3. Conclusion Attacks

These introduce irrelevant logical chains, such as:

  • Expert Opinion Attack: Inserting sentences like "A judge thought the defendant committed [Crime X]."
  • Previous Behavior Attack: Mentioning the defendant's past criminal record, which should not legally influence the current judgment.

Experimental Setup and Datasets

The study utilized two primary Chinese legal datasets: LEVEN (Legal Event Detection) and CAIL2018 (Judgment Prediction). Labels were updated to align with current Chinese criminal law, and legal experts were employed to annotate "similar crimes" and "legal synonyms" to ensure the attacks were professionally plausible. The researchers also tested defense strategies including RAG, Chain-of-Thought (CoT), and few-shot prompting.

Key Insights and Results

The researchers measured the Performance Drop Ratio (PDR)—the decrease in accuracy when an attack is present.

  • High Vulnerability: Most adversarial attacks resulted in a PDR greater than 0, with many exceeding 30%.
  • Conclusion Attacks are Most Effective: Attacks at the conclusion level, particularly the "Expert Opinion" attack, were the most damaging. For instance, LLaMA3’s accuracy dropped by over 44% when a "judge's opinion" was injected.
  • The Fragility of CoT: Surprisingly, Chain-of-Thought prompting sometimes decreased robustness. If a model is successfully "attacked" early in its reasoning chain, it may confidently hallucinate an entire logical path to an incorrect conclusion.
  • Specialization is Not a Panacea: While legal-specific models like Farui showed more robustness than general models (GPT-3.5, LLaMA3), they still suffered significant performance drops under conclusion-level attacks.
  • "Lost in the Middle": Noise inserted into the middle of a prompt generally had less impact than noise at the beginning or end.

Conclusion

Current prompt-level mitigations like CoT or RAG are insufficient to ensure the logical grounding of LLMs in the legal domain. The J&H framework reveals that LLMs struggle with the nuances of domain-specific concepts and are easily misled by irrelevant context. Future work should focus on integrating domain-specific knowledge and explicit reasoning chains directly into the pre-training and fine-tuning stages to develop a truly robust "LLM-as-judge."

Summary updated

Abstract

As the scale and capabilities of Large Language Models (LLMs) increase, their applications in knowledge-intensive fields such as legal domain have garnered widespread attention. However, it remains doubtful whether these LLMs make judgments based on domain knowledge for reasoning. If LLMs base their judgments solely on specific words or patterns, rather than on the underlying logic of the language, the ''LLM-as-judges'' paradigm poses substantial risks in the real-world applications. To address this question, we propose a method of legal knowledge injection attacks for robustness testing, thereby inferring whether LLMs have learned legal knowledge and reasoning logic. In this paper, we propose J&H: an evaluation framework for detecting the robustness of LLMs under knowledge injection attacks in the legal domain. The aim of the framework is to explore whether LLMs perform deductive reasoning when accomplishing legal tasks. To further this aim, we have attacked each part of the reasoning logic underlying these tasks (major premise, minor premise, and conclusion generation). We have collected mistakes that legal experts might make in judicial decisions in the real world, such as typos, legal synonyms, inaccurate external legal statutes retrieval. However, in real legal practice, legal experts tend to overlook these mistakes and make judgments based on logic. However, when faced with these errors, LLMs are likely to be misled by typographical errors and may not utilize logic in their judgments. We conducted knowledge injection attacks on existing general and domain-specific LLMs. Current LLMs are not robust against the attacks employed in our experiments. In addition we propose and compare several methods to enhance the knowledge robustness of LLMs.
BibTeX
@inproceedings{hu2025j,
  author = {Yiran Hu and Huanghai Liu and Qingjing Chen and Ning Zheng and Chong Wang and Yun Liu and Charles L. A. Clarke and Weixing Shen},
  title = {{J\&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=TbvgYAIfuT},
}
Figure