ClosureBench: A Benchmark for Reasoning under Closure Assumptions

Francesco Belardinelli, Flavio Corradini, Matteo LeonesiarXiv 2026 · arXiv

TL;DR

ClosureBench is a new benchmark designed to test whether Large Language Models (LLMs) can follow explicit "closure contracts"—the logical rules that dictate whether missing information should be treated as unknown or false. The study reveals that while many models appear accurate on the surface, they often fail to adjust their reasoning when these assumptions change, particularly struggling with the Local Closed World Assumption (LCWA).

What Problem Does This Paper Address?

In real-world deployments, AI agents rarely have perfect information. When a record is silent about a specific fact, how should an agent react?

In symbolic logic, this is handled by "closure assumptions":

  • Open World Assumption (OWA): If it’s not in the database, it’s unknown.
  • Closed World Assumption (CWA): If it’s not in the database, it’s false.
  • Local Closed World Assumption (LCWA): Only specific, "complete" parts of the database treat missing info as false; everything else is unknown.

Current benchmarks often conflate these logical assumptions with general reasoning abilities. If a model fails, is it because it can't reason, or because it's using the wrong "contract" for missing data? ClosureBench isolates this variable to see if LLMs can strictly follow the closure rules provided in a prompt.

Methods

The authors developed a dataset where each "base scenario" (a set of facts and rules) is tested under three different variants: OWA, CWA, and LCWA. Crucially, the facts and rules remain identical across variants; only the "closure contract" in the prompt changes.

To pass a scenario, a model must provide the correct answer for all three variants. This leads to a new metric: Closure Switch Accuracy. This prevents models from "getting lucky" by defaulting to a single assumption (like always assuming missing info is "unknown").

The paper also introduces three "Agent Extensions" to see how these logical errors translate into behavior:

  1. Ask/Act: Does the agent correctly decide to "deny" (false) or "request more info" (unknown)?
  2. SourceScope: Can the agent distinguish between a "complete" source and a "partial" one?
  3. ClosureShift: Can the agent update its answer across multiple turns as the completeness of the data changes?

Datasets

The primary contribution is ClosureBench, which consists of 960 English instances derived from 320 base scenarios across eight domains (e.g., cloud deployment, finance controls, robotics operations).

As shown in the comparison table below, ClosureBench is unique because it is the only suite that keeps the symbolic case fixed while specifically targeting the "scenario switch" and LCWA scope.

Comparison of ClosureBench with related logic and agent benchmarks.

Key Insights and Ideas

  • The "Unknown" Bias: Many models have a strong internal bias toward OWA. Even when a prompt explicitly states that a predicate is "complete" (meaning missing info = false), models often default to "unknown."
  • Accuracy is a Mask: A model might have 90% overall accuracy but 0% "core switch accuracy." This happens when a model correctly answers "true" or "false" based on explicit facts but fails every time the answer depends on the closure contract.
  • LCWA is the Hardest: Models struggle significantly with "local" closure. They find it difficult to treat some missing facts as false while simultaneously treating others as unknown within the same context.
  • Logical Errors = Action Failures: In the Ask/Act extension, models frequently exhibited "Excessive Ask"—requesting more information from a user even when the "complete" record already provided enough evidence to justify a denial.

Key Results

The results show a massive gap between general performance and closure-specific reasoning. For instance, Mistral Medium achieves a respectable 95.59% overall accuracy, but its "Core Switch" accuracy (measuring only the cases where the label actually changes with the contract) drops to 78.47%.

Table 2: Base ClosureBench results over three runs with temperature 0.

In the Ask/Act extension, the "Excessive Ask" metric highlights a major behavioral flaw. Llama 4 Scout, for example, requested unnecessary information in over 45% of cases where it should have simply denied the request based on the closure contract.

Table 3: Results for the Ask/Act extension, showing how closure errors lead to unnecessary requests for information.

Notably, Llama 4 Maverick served as a "positive control," nearly solving the suite with 99.86% accuracy, suggesting that the benchmark is logically sound and solvable by state-of-the-art models.

Limitations and Future Work

The authors acknowledge that ClosureBench is template-generated, which ensures logical auditability but limits linguistic diversity. It doesn't test how models handle naturalistic paraphrases or "messy" real-world language.

Furthermore, the agent extensions are "controlled experiments"—the models don't actually execute tools or interact with a live environment. Future work could involve integrating these closure assumptions into more complex, multi-agent environments where different agents operate under different (and perhaps unstated) closure contracts.

Summary updated

Abstract

AI agents built on large language models (LLMs) often answer questions from incomplete records. Missing information might be interpreted as unknown or false, depending on the completeness of the source. Current evaluations fail to distinguish these two aspects, as they are usually interwoven with other reasoning demands. We present ClosureBench, a benchmark that tests whether LLMs follow the stated rules for dealing with missing information. Every base scenario is tested under OWA, CWA, and LCWA, and only the treatment of missing information changes between variants. ClosureBench provides a verified base benchmark as well as agent extensions for action choice, source scope, and updates. A model is considered correct on a base scenario only if it provides correct answers in all of its closure variants. Across 960 instances, overall accuracy can mask substantial closure mistakes. Models achieve overall accuracy from 71.18% to 99.86% across replicated runs, but their core switch accuracy, measured on the scenarios whose gold label actually changes with the contract, varies between 18.23% and 99.31%. Large discrepancies emerge under LCWA. For example, some agents choose unknown despite the prompt indicating that missing information should be treated as false. Our findings demonstrate that evaluations of AI agents should not only measure whether agents can reason over existing facts but also whether they use the appropriate assumption when information is absent.
BibTeX
@misc{belardinelli2026closurebench,
  author = {Francesco Belardinelli and Flavio Corradini and Matteo Leonesi},
  title = {{ClosureBench: A Benchmark for Reasoning under Closure Assumptions}},
  year = {2026},
  eprint = {2611.26335},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26335},
}
Figure