LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appeal

Michal Karp, Anna Kubaszewska, Magdalena Król, Robert Król, Aleksander Smywinski-Pohl, Mateusz Szymanski, Witold WydmanskiarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) become increasingly sophisticated, their application in the legal domain has transitioned from simple document summarization to more complex tasks like legal reasoning and adjudication. However, the high-stakes nature of public procurement disputes—specifically within the jurisdiction of the Polish National Appeal Chamber (NAC)—requires a level of precision that goes beyond mere linguistic fluency. This study evaluates whether state-of-the-art LLMs can pass the rigorous qualifying exam for NAC membership and, crucially, whether the "LLM-as-a-judge" paradigm is a reliable metric for evaluating legal competence.

Methodology

The researchers developed a hybrid evaluation pipeline to test three models: GPT-4.1, Claude 4 Sonnet, and Bielik-11B-v2.6 (a Polish-native model). The evaluation was structured to mirror the official recruitment process for the NAC.

The Pipeline

To support the models, an advanced Retrieval-Augmented Generation (RAG) system was implemented. This system utilized a vectorized database containing the Polish Public Procurement Law (PZP), the Civil Code, relevant judicial decisions, and official legal commentaries.

The Examination

The models were subjected to a two-part examination:

  1. Knowledge Test: A 50-question multiple-choice test focusing on statutory knowledge.
  2. Practical Written Exam: A complex task requiring the drafting of a formal legal judgment based on a specific factual scenario involving public procurement disputes.

Dual Evaluation Framework

A critical component of the study was the comparison of two different grading entities for the written exam:

  • The Official Jury: A panel of human legal experts from the NAC.
  • LLM-as-a-Judge: GPT-4o, provided with a detailed rubric based on official NAC standards.

Datasets

The study leveraged a comprehensive legal corpus, including:

  • Historical Data: 250 questions and five case studies from previous NAC exams.
  • The 2025 Exam: 50 new questions and a fresh case study from the most recent recruitment cycle.
  • Validation Set: 101 manually curated cases designed to test the extraction of formal legal criteria, such as filing deadlines and fee calculations.

Key Results

The Knowledge Test

The models showed competent performance on the multiple-choice section. GPT-4.1 achieved an accuracy of 88.4% on historical questions when supported by RAG. However, performance dipped to 74% on the live 2025 exam, suggesting that "unseen" or more contemporary legal nuances remain challenging.

The Written Exam: A Universal Failure

Despite their performance on the knowledge test, all models failed the practical written exam. The passing threshold is 50 out of 100 points. The human jury scores were as follows:

  • GPT-4.1: 37/100 (The highest score, yet still legally flawed).
  • Claude 4 Sonnet: 30/100 (Described as a "patchwork of mismatched fragments").
  • Bielik-11B: 8/100 (Resulted in outright disqualification).

The Failure of LLM-as-a-Judge

The most significant finding was the extreme divergence between human and AI grading. While the human jury identified "cardinal failures" in the AI-generated judgments, the LLM judge (GPT-4o) was remarkably lenient. GPT-4o awarded scores between 85 and 90 out of 100 to all models. Furthermore, while humans saw a 29-point performance gap between the best and worst models, the AI judge saw only a 5-point difference. This suggests that LLM judges are easily swayed by stylistic polish and professional-sounding prose, failing to detect substantive legal errors.

Discussion and Insights

Interpretive vs. Mechanical Tasks

The study highlighted a clear divide in AI capabilities. LLMs were highly reliable (97%+ accuracy) at "mechanical" tasks, such as extracting filing dates or verifying if a fee was paid. However, they struggled with "interpretive" tasks—the core of legal work—which involve applying abstract laws to nuanced, specific facts.

Hallucinations and Structural Errors

Even with RAG, the models frequently "hallucinated" legal provisions, citing non-existent articles like "Article 514(4) PZP." More critically, the models failed on basic procedural requirements. For instance, all models failed to correctly identify the required composition of the adjudicating panel, proposing a three-member panel for a case that legally required a single judge. In a real-world setting, such an error would lead to the immediate invalidation of the judgment.

Conclusion

The study concludes that while LLMs are valuable tools for mechanical augmentation—potentially offering 30-40% efficiency gains in administrative tasks—they are currently unfit for autonomous adjudication. The "LLM-as-a-judge" approach is particularly dangerous in specialized legal contexts, as it lacks the critical eye necessary to distinguish between a well-formatted document and a legally sound one.

Future work will focus on supervised fine-tuning (SFT) using specific Polish judicial rulings and the development of multi-agent systems where specialized agents handle different aspects of a judgment (e.g., fact-finding vs. statutory interpretation) to mitigate the risk of logical and procedural failures.

Summary updated

BibTeX
@misc{karp2025llm,
  author = {Michal Karp and Anna Kubaszewska and Magdalena Król and Robert Król and Aleksander Smywinski-Pohl and Mateusz Szymanski and Witold Wydmanski},
  title = {{LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appeal}},
  year = {2025},
  eprint = {2511.04205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.04205},
}
Figure