TL;DR
BMMS (Behavioral Misalignment Metric Suite) shifts AI evaluation from fixed categories like "toxicity" to policy-conditioned measurement, allowing users to test models against any specific rulebook (e.g., the EU AI Act or internal company policies). The study reveals a striking gap in vendor self-compliance: while Claude models largely follow their own "Constitution," GPT-4o fails over 70% of the hard constraints defined in OpenAI’s own Model Spec.
What Problem Does This Paper Address?
Current LLM benchmarks use a "fixed taxonomy"—they measure predefined dimensions like hallucination, sycophancy, or toxicity. However, an operator in a regulated industry (like a bank or a healthcare provider) doesn't just need to know if a model is "generally good"; they need to know if it complies with specific, codified rules like FINRA suitability or the EU AI Act.
The authors identify this as a missing conditioning problem. Existing benchmarks are silent on specific rulebooks because those dimensions were never built into the test. BMMS addresses this by treating the policy corpus itself as an input argument, creating a customizable measurement layer that can be applied to any system that communicates in natural language.
Methods
The BMMS pipeline (illustrated in Figure 1) transforms a raw policy into a "compliance profile":
- Extraction & Admission: Obligations are extracted from documents. A "bindability panel" of three LLMs determines if an AI assistant can actually comply with the rule in a single response (filtering out firm-level duties).
- Probe Generation: For every admitted rule, the suite generates a "pressure ladder" of four probes ($L_0$ to $L_3$): direct requests, embedded scenarios, boundary cases, and adversarial pressure.
- Measurement: Targets (the models being tested) answer the probes. A "Pinned Judge" (Claude-3-Opus) scores the responses.
- The Admission Test: To ensure the metrics are scientifically sound, the authors propose five criteria (Construct, Independence, Sensitivity, Discrimination, and Attribution). Crucially, the Attribution check (A5) ensures that the metric measures the model's behavior, not just the judge's biases, by re-scoring results with different "swap judges."
Datasets
The authors evaluated 8 models from 4 providers against 1,003 obligations extracted from 11 public documents across five "registers":
- Vendor Specifications: OpenAI Model Spec, Claude’s Constitution.
- Conduct Regulation: FINRA 2026 guidance, MiFID II.
- Analytical Guidance: BIS reports.
- Statutes: EU AI Act, Colorado AI Act, CA SB 1001.
- Acceptable-use Policy (AUP): FS-ISAC GenAI AUP.
The final "frozen probe set" used for the core results consisted of 397 majority-bindable rules.
Key Insights and Ideas
- The "Judge Swap" Revelation: Absolute compliance scores can vary by as much as 0.22 depending on which LLM is acting as the judge. However, the relative ordering of models remains almost perfectly stable (Kendall $\tau = 1.000$). This suggests that while we can't yet trust absolute "compliance percentages" as objective truth, we can reliably rank models.
- Document Difficulty vs. Favoritism: Models often score better on their own vendor's specifications. However, the authors found this isn't necessarily "favoritism." By testing unrelated models (like Llama 3.1) on the same specs, they showed that some rulebooks (like Claude's Constitution) are simply "easier" to follow than others (like OpenAI's Model Spec).
- Capability $\neq$ Compliance: Compliance scores do not monotonically track with standard capability benchmarks. A smaller, "distilled" model might follow a specific rulebook better than a massive frontier model, proving that compliance is a distinct behavioral property.
Key Results
The suite produces a multi-dimensional profile rather than a single score (see Figure 4).
- Compliance Attainment Score (CAS): GPT-4o showed a significant failure rate on its own vendor's rules. It violated 70.7% of the hard constraints in the OpenAI Model Spec, whereas Claude models failed only 0.0% to 9.1% of their own specifications.
- Compliance Robustness Margin (CRM): This measures how much compliance drops under adversarial pressure. GPT-4o’s compliance degraded by 0.213 when moving from polite requests to adversarial framing, while Claude models remained nearly "flat," showing high robustness.
- Register Spread: Models generally performed best on vendor specs and worst on transparency statutes and analytical guidance (see Figure 2).
Limitations and Future Work
- Human Calibration: The validation study relied on a single domain expert for 60 items. While precision was high for vendor specs (94.4%), it dropped for analytical guidance (50.0%), suggesting the "bindability" filter needs refinement for complex regulatory prose.
- Extraction Bias: The current extraction process tends to shed "qualifiers" (exemptions) from rules. This makes the probes "harder" than the actual rules, meaning the reported violation rates are likely conservative (inflated).
- Unmeasured Dimensions: Two dimensions—Behavioral Invariance (BIV) and Behavioral Disposition Index (BDI)—are defined in the suite but were not measured at full scale in this paper. Full implementation of these is a clear path for future work.
Summary updated