CHANCERY: Evaluating Corporate Governance Reasoning Capabilities in Language Models

Lucas Irwin, Arda Kaz, Peiyao Sheng, Sewoong Oh, Pramod ViswanatharXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) increasingly permeate professional workflows, their ability to handle complex, high-stakes reasoning tasks remains a critical frontier. While these models excel at basic legal tasks such as summarization and entity extraction, they often falter when faced with "ratiocination"—the multi-step deductive logic required to apply specific rules to complex facts. Researchers from Sentient Labs have introduced CHANCERY, a novel benchmark designed to evaluate whether LLMs can navigate the intricate world of corporate governance.

The Challenge of Corporate Ratiocination

Corporate governance serves as an ideal stress test for logical reasoning. Determining the legality of an executive action requires a model to ingest a corporate "charter" (a foundational set of governing rules), evaluate a proposed action, and cross-reference it with both the charter and broader legal frameworks, such as the Delaware General Corporation Law (DGCL).

Unlike existing legal benchmarks that focus on simple classification or clause identification, CHANCERY requires normative reasoning. Models must move beyond pattern matching to demonstrate a true understanding of how governing provisions interact.

The CHANCERY Benchmark

The CHANCERY dataset is a binary classification task where models must determine if a specific proposal—such as a shareholder's attempt to introduce cumulative voting—is legal under a given charter.

An example question from the CHANCERY benchmark involving cumulative voting.

Dataset Composition

The benchmark consists of:

  • 502 Handcrafted Questions: Scenarios designed to be realistic and legally sound.
  • 79 Diverse Corporate Charters: Selected from a pool of 10,000 real-world documents across industries like Manufacturing, Tech, and Finance.
  • 24 Core Governance Principles: Including Poison Pills, Golden Parachutes, and Anti-greenmail provisions.

To ensure high quality, the scenarios were iteratively generated using GPT-4o and then heavily refined and hand-labeled by human experts to establish a "gold standard" ground truth.

Experimental Results and the "Reasoning Gap"

The evaluation of state-of-the-art (SOTA) models revealed a significant gap between general linguistic capability and specialized legal reasoning.

The Reasoning Model Paradox

One of the most surprising findings was that specialized reasoning models, such as DeepSeek-R1 and Claude 3.7 Sonnet, often underperformed compared to general-purpose models like GPT-4o. This suggests that current "reasoning" optimizations may not yet capture the domain-specific nuances and linguistic precision required for corporate law.

A comparison between a correct legal answer and the reasoning failure of DeepSeek-R1.

Agents vs. Models

The study found that agentic frameworks significantly outperformed standalone models. By using ReAct (Reasoning-and-Action) or CodeAct (which translates logic into executable code), models were better able to structure their reasoning and avoid common "hallucination" traps. The CodeAct agent achieved the highest accuracy at 78.1%.

Model / Agent Accuracy
CodeAct Agent 78.1%
GPT-4o 75.2%
Llama 3.1-70B 72.7%
Claude 3.7-Sonnet 64.5%
DeepSeek-R1 58.6%
QwQ-32B 55.4%

Key Insights and Blind Spots

The researchers identified several specific hurdles for LLMs:

  1. The Multi-Hop Hurdle: Performance dropped by approximately 15% when questions required "multi-hop" reasoning—connecting multiple disparate sections of a charter to reach a conclusion.
  2. Implicit Legal Standards: Models struggled when a principle was not explicitly stated in the charter but was implied by broader legal standards. For instance, DeepSeek-R1 scored only 27.6% on Anti-greenmail provisions.
  3. Reliability Gap: Even the best-performing agent (78.1%) falls short of the near-perfect accuracy required for autonomous legal decision-making, highlighting the continued necessity of human oversight.

Limitations and Future Directions

While CHANCERY sets a new standard for legal AI evaluation, it currently focuses on US-centric corporate law (specifically Delaware). Future iterations could expand to international jurisdictions and explore more "nebulous" governance structures, such as Distributed Autonomous Organizations (DAOs), which lack established legal precedents.

By open-sourcing the CHANCERY benchmark, the researchers provide a rigorous framework for developing the next generation of reasoning-capable AI in the legal domain.

Summary updated

Abstract

Law has long been a domain that has been popular in natural language processing (NLP) applications. Reasoning (ratiocination and the ability to make connections to precedent) is a core part of the practice of the law in the real world. Nevertheless, while multiple legal datasets exist, none have thus far focused specifically on reasoning tasks. We focus on a specific aspect of the legal landscape by introducing a corporate governance reasoning benchmark (CHANCERY) to test a model's ability to reason about whether executive/board/shareholder's proposed actions are consistent with corporate governance charters. This benchmark introduces a first-of-its-kind corporate governance reasoning test for language models - modeled after real world corporate governance law. The benchmark consists of a corporate charter (a set of governing covenants) and a proposal for executive action. The model's task is one of binary classification: reason about whether the action is consistent with the rules contained within the charter. We create the benchmark following established principles of corporate governance - 24 concrete corporate governance principles established in and 79 real life corporate charters selected to represent diverse industries from a total dataset of 10k real life corporate charters. Evaluations on state-of-the-art (SOTA) reasoning models confirm the difficulty of the benchmark, with models such as Claude 3.7 Sonnet and GPT-4o achieving 64.5% and 75.2% accuracy respectively. Reasoning agents exhibit superior performance, with agents based on the ReAct and CodeAct frameworks scoring 76.1% and 78.1% respectively, further confirming the advanced legal reasoning capabilities required to score highly on the benchmark. We also conduct an analysis of the types of questions which current reasoning models struggle on, revealing insights into the legal reasoning capabilities of SOTA models.
BibTeX
@misc{irwin2025chancery,
  author = {Lucas Irwin and Arda Kaz and Peiyao Sheng and Sewoong Oh and Pramod Viswanath},
  title = {{CHANCERY: Evaluating Corporate Governance Reasoning Capabilities in Language Models}},
  year = {2025},
  eprint = {2506.04636},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.04636},
}
Figure