BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models

Anthony Rhodes, Gadi SingerarXiv 2026 · arXiv

TL;DR

BMMS (Behavioral Misalignment Metric Suite) shifts AI evaluation from fixed categories like "toxicity" to policy-conditioned measurement, allowing users to test models against any specific rulebook (e.g., the EU AI Act or internal company policies). The study reveals a striking gap in vendor self-compliance: while Claude models largely follow their own "Constitution," GPT-4o fails over 70% of the hard constraints defined in OpenAI’s own Model Spec.

What Problem Does This Paper Address?

Current LLM benchmarks use a "fixed taxonomy"—they measure predefined dimensions like hallucination, sycophancy, or toxicity. However, an operator in a regulated industry (like a bank or a healthcare provider) doesn't just need to know if a model is "generally good"; they need to know if it complies with specific, codified rules like FINRA suitability or the EU AI Act.

The authors identify this as a missing conditioning problem. Existing benchmarks are silent on specific rulebooks because those dimensions were never built into the test. BMMS addresses this by treating the policy corpus itself as an input argument, creating a customizable measurement layer that can be applied to any system that communicates in natural language.

Methods

The BMMS pipeline (illustrated in Figure 1) transforms a raw policy into a "compliance profile":

  1. Extraction & Admission: Obligations are extracted from documents. A "bindability panel" of three LLMs determines if an AI assistant can actually comply with the rule in a single response (filtering out firm-level duties).
  2. Probe Generation: For every admitted rule, the suite generates a "pressure ladder" of four probes ($L_0$ to $L_3$): direct requests, embedded scenarios, boundary cases, and adversarial pressure.
  3. Measurement: Targets (the models being tested) answer the probes. A "Pinned Judge" (Claude-3-Opus) scores the responses.
  4. The Admission Test: To ensure the metrics are scientifically sound, the authors propose five criteria (Construct, Independence, Sensitivity, Discrimination, and Attribution). Crucially, the Attribution check (A5) ensures that the metric measures the model's behavior, not just the judge's biases, by re-scoring results with different "swap judges."
The BMMS pipeline and metric suite components.

Datasets

The authors evaluated 8 models from 4 providers against 1,003 obligations extracted from 11 public documents across five "registers":

  • Vendor Specifications: OpenAI Model Spec, Claude’s Constitution.
  • Conduct Regulation: FINRA 2026 guidance, MiFID II.
  • Analytical Guidance: BIS reports.
  • Statutes: EU AI Act, Colorado AI Act, CA SB 1001.
  • Acceptable-use Policy (AUP): FS-ISAC GenAI AUP.

The final "frozen probe set" used for the core results consisted of 397 majority-bindable rules.

Key Insights and Ideas

  • The "Judge Swap" Revelation: Absolute compliance scores can vary by as much as 0.22 depending on which LLM is acting as the judge. However, the relative ordering of models remains almost perfectly stable (Kendall $\tau = 1.000$). This suggests that while we can't yet trust absolute "compliance percentages" as objective truth, we can reliably rank models.
  • Document Difficulty vs. Favoritism: Models often score better on their own vendor's specifications. However, the authors found this isn't necessarily "favoritism." By testing unrelated models (like Llama 3.1) on the same specs, they showed that some rulebooks (like Claude's Constitution) are simply "easier" to follow than others (like OpenAI's Model Spec).
  • Capability $\neq$ Compliance: Compliance scores do not monotonically track with standard capability benchmarks. A smaller, "distilled" model might follow a specific rulebook better than a massive frontier model, proving that compliance is a distinct behavioral property.

Key Results

The suite produces a multi-dimensional profile rather than a single score (see Figure 4).

  • Compliance Attainment Score (CAS): GPT-4o showed a significant failure rate on its own vendor's rules. It violated 70.7% of the hard constraints in the OpenAI Model Spec, whereas Claude models failed only 0.0% to 9.1% of their own specifications.
  • Compliance Robustness Margin (CRM): This measures how much compliance drops under adversarial pressure. GPT-4o’s compliance degraded by 0.213 when moving from polite requests to adversarial framing, while Claude models remained nearly "flat," showing high robustness.
  • Register Spread: Models generally performed best on vendor specs and worst on transparency statutes and analytical guidance (see Figure 2).
The BMMS profile for three frontier models, showing performance across six axes.

Limitations and Future Work

  • Human Calibration: The validation study relied on a single domain expert for 60 items. While precision was high for vendor specs (94.4%), it dropped for analytical guidance (50.0%), suggesting the "bindability" filter needs refinement for complex regulatory prose.
  • Extraction Bias: The current extraction process tends to shed "qualifiers" (exemptions) from rules. This makes the probes "harder" than the actual rules, meaning the reported violation rates are likely conservative (inflated).
  • Unmeasured Dimensions: Two dimensions—Behavioral Invariance (BIV) and Behavioral Disposition Index (BDI)—are defined in the suite but were not measured at full scale in this paper. Full implementation of these is a clear path for future work.

Summary updated

Abstract

Behavioral evaluation of language models fixes its dimensions in advance: hallucination, sycophancy, toxicity, over-refusal. An operator in a regulated industry needs a different question answered: does a system do what a particular rulebook requires? We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument and returning a compliance profile over it, for any published policy and any system that answers in natural language. The Compliance Attainment Score (CAS) measures severity-weighted attainment over the obligations that bind the system, and the Compliance Robustness Margin (CRM) the share of that attainment surviving graded adversarial pressure; Behavioral Invariance (BIV) and the Behavioral Disposition Index (BDI) are specified and released with the suite. Dimensions enter only by passing a published admission test whose binding criterion is attribution: a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument. Applying it excluded one of the quantities the suite itself reported. Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance, extraction fidelity is 94.6%, and re-scoring under judges from two other providers leaves target ordering invariant while the absolute score of a single target spans up to 0.22. We apply it to the two most detailed published behavioral specifications. Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter: GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.
BibTeX
@misc{rhodes2026bmms,
  author = {Anthony Rhodes and Gadi Singer},
  title = {{BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models}},
  year = {2026},
  eprint = {2611.09147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09147},
}
Figure