Reading the Legal Force of Voluntary AI Commitments

When an AI company writes "we will red-team all frontier models before release," it has made a promise someone could check. When it writes "we are committed to exploring approaches to responsible evaluation where appropriate," it has said almost nothing. Current audits of voluntary AI commitments cannot tell these two sentences apart. Researchers read public documents by hand and mark each indicator as present or absent. A firm that hedges gets the same credit as one that binds itself. This proposal asks whether language models that were further trained on legal and regulatory text, and so have read billions of tokens of contracts, statutes, and Securities and Exchange Commission (SEC) filings, can grade how much a disclosure actually commits the firm. It then uses that grade to answer a question that has gone untested: does the reputational benefit firms seek from voluntary signaling come from substantive promises or from hedged coverage?

The core claim is narrow and falsifiable. Telling a binding commitment from a hedged one is a legal-reading skill. Its cues include modal verbs, conditions, safe-harbor disclaimers, quantifiers, and whether a claim can be verified. That makes it exactly the kind of skill domain pre-training should help with, if it helps anywhere, whereas merely detecting whether a topic is mentioned is not. We test this with a matched pair: a legal continually-pretrained model against its own general base model, on a new graded "commitment strength" dataset built from the public evidence behind existing AI disclosure audits.

What is already known, and where the gap is

Two manual audits supply both the motivation and the raw material. Wang et al. turned the 2023 White House voluntary commitments into 30 binary indicators and audited 16 companies. The average score was 52%. Frontier Model Forum (the industry safety coalition) members averaged 74.4% versus 40.7% for non-members. The authors explicitly flag "vacuous success" and underspecified language that allows nominal compliance. Bommasani et al. built the Foundation Model Transparency Index, 100 binary indicators across 10 developers with a mean of 37/100. They note that disclosure is not accountability and that binary scoring misses depth. Both papers identify the problem we target, and both stop at binary presence. We keep their indicators and their evidence passages and replace the 0/1 judgment with a graded one.

The closest automated work is EulerESG. It uses dense-plus-keyword retrieval and prompts general frontier models to label ESG report content against SASB standards as disclosed, partially disclosed, or not disclosed. GPT-5 reaches 0.95 average accuracy. That result constrains our design. It suggests that presence detection in corporate reports is close to solved by general models. Automating the FMTI or White House rubric with a legal model would therefore be the obvious combination, and probably an uninteresting one. We deliberately target the dimension EulerESG does not measure: how strongly the present text commits its author. SEC-QA builds question answering over 10-K, 10-Q, and 8-K filings and shows that code-generating navigation beats vanilla retrieval. Its questions are numerical. It supplies a retrieval lesson, that boilerplate-heavy filings confuse vector search, but it says nothing about qualitative commitments.

On the modeling side, SaulLM-54B and SaulLM-141B continually pre-train Mixtral mixture-of-experts models on roughly 520B legal tokens, including SEC EDGAR. SaulLM-141B reaches 75.34% mean balanced accuracy on LegalBench-Instruct against 73.35% for GPT-4. The same paper reports degraded general instruction-following and inverse scaling on about 20% of tasks. These caveats shape how we evaluate the model. Hendrycks et al. are the main counterweight. On CUAD contract clause extraction, 8GB of extra legal pre-training gave only about 3% improvement once expert labels and a strong architecture (DeBERTa-xlarge) were present. That is a real reason to expect a null result, and the proposal is designed to detect one cleanly. LegalBench supplies two further facts. Long-form disclosure tasks such as Supply Chain Disclosures sit near 50% for all tested models. General models also show a "legalese penalty": GPT-3.5 does worse on technical legal phrasing than on plain English. LexGLUE shows Legal-BERT winning overall at 79.8 µ-F1, but also that TF-IDF with a linear classifier beats transformers on long SCOTUS documents. We take that as a warning that surface statistics can be strong on diffuse corporate text.

Research Program

The first step is a dataset. For each of the 30 White House indicators and a subset of roughly 40 FMTI indicators, we collect the passages auditors would cite as evidence from model cards, system cards, safety frameworks, blog posts, and bug-bounty pages for the audited companies. We add matched passages from public ESG reports of the companies EulerESG studied, which gives an out-of-domain check. Annotators with legal training score each passage on a five-level commitment-strength scale. The levels run from silence, through topical mention with no undertaking, hedged or conditional intent ("may," "where appropriate," "we aim to"), and unconditional but unverifiable undertaking, up to specific, verifiable commitment with scope, timing, or measurable criteria. Annotators also mark the spans responsible for downgrading a passage, such as the hedge, the condition, or the disclaimer. We report inter-annotator agreement (weighted kappa) and treat agreement below moderate as a sign the construct is ill-posed. If that happens, the project would revise the scale before modeling.

Because naturally occurring passages confound hedging with topic and company style, we also build a controlled minimal-pair set. Lawyers edit real passages to change only the legal force, for example replacing "will" with "intends to," adding a "subject to applicable law" carve-out, or deleting a quantifier. Each pair has a known direction of change. This set isolates the specific competence we claim domain pre-training provides. A parallel plain-English rewrite of each passage tests whether any advantage of the legal model is just immunity to the legalese penalty or reflects something deeper.

The modeling comparison is built around one matched pair: SaulLM-54B against Mixtral 8x7B, the model it was continually pre-trained from. They share architecture and size and differ only in legal pre-training. SaulLM is known to follow generic instructions less well, so we evaluate each model in two ways. The first is zero- and few-shot prompting with fixed, pre-registered demonstrations; LegalBench reports that demonstration choice alone can swing accuracy by over 20 points. The second is a lightweight ordinal head or low-rank adapter trained on frozen representations with an identical budget. The second route asks whether legal knowledge is present in the weights even when the instruction interface is degraded. We then run the selected model across all audited companies and re-score the White House and FMTI results as graded profiles.

The substantive payoff is a signaling-theory test. Wang et al. found that coalition members score far higher on presence. We ask whether that gap survives once credit is weighted by commitment strength, or whether coalition membership mainly buys broader, hedged coverage. The broader signaling literature, for example the certification-driven disclosure expansion documented by Zhang et al. and the strategic use of self-regulation to forestall binding rules described by Anderson, predicts these two readings differently. Expansion to protect reputational capital should raise coverage. Credible differentiation from low-quality actors, in the sense of Dong, requires costly, verifiable content. We also test the security category directly. Model-weight security averaged 17%, with 11 of 16 companies at zero, so we can check whether the few non-zero disclosures are substantive or nominal.

Baselines

The naive reference point is a lexical hedge scorer: a hand-written lexicon of modal and conditional markers and disclaimer phrases, plus a TF-IDF linear ordinal classifier trained on the same labels. LexGLUE's SCOTUS result shows this family can beat transformers on long, diffuse documents, so it is not a strawman. Any model that fails to beat it on the natural passages has learned nothing beyond word lists. A second surface baseline scores boilerplate by measuring the text similarity of a passage to the same company's earlier documents and to peer companies' documents. Recycled language is a plausible proxy for vacuity, and if this baseline matches the neural models, the "legal reading" story collapses.

The strongest existing method is the EulerESG recipe (EulerESG): dense plus keyword retrieval followed by prompting a frontier general model. We extend its three-way label to our five-level scale with the same rubric text every other model sees. Its reported figure is 0.95 average accuracy (GPT-5, three-way disclosure classification). On our graded task, there is no reported result found on Lacuna for commitment-strength grading by any model. For legal reasoning more broadly, the reference figures are SaulLM-141B at 75.34% versus GPT-4 at 73.35% mean balanced accuracy on LegalBench-Instruct (Colombo et al.). A supervised strong general encoder, DeBERTa-xlarge fine-tuned on our labels, follows Hendrycks et al.'s finding that supervision plus architecture captures most of the gain on CUAD.

The key ablation is Mixtral 8x7B versus SaulLM-54B under both the prompting and frozen-representation protocols. To keep the comparison fair, all fine-tuned models get the same training split, the same hyperparameter search budget, and the same number of training passes. All prompted models get identical rubrics and demonstrations. Retrieval is held fixed across models, using gold evidence passages, so that reading is compared rather than finding. Beating the field means two things. First, SaulLM must exceed Mixtral on the minimal-pair directional accuracy and on weighted kappa against annotators by a margin larger than the seed-to-seed and demonstration-to-demonstration variance. Second, it must at least match the frontier-model prompting baseline on the natural passages at a fraction of the inference cost.

Why not just…?

Why not just prompt a frontier general model, as EulerESG does? This is the most serious threat. EulerESG's 0.95 suggests frontier models read corporate reports well, and they may grade hedging well too. The evidence that they might not is indirect: LegalBench's legalese penalty and the ~50% ceiling on long-form disclosure tasks. Neither directly tests hedged commitments. The minimal-pair set settles it. If the frontier model tracks every hedge edit as well as SaulLM does, we report that the general model suffices and that domain pre-training is unnecessary for this task. The program's audit re-scoring would then proceed with the general model.

Why not just fine-tune a strong general encoder on expert labels? Hendrycks et al. found that this captures nearly all the gain on contract clauses, with legal pre-training adding about 3%. Our task differs in that the cues are subtle reversals of legal force rather than clause topics, but that is a hypothesis, not evidence. The DeBERTa-xlarge baseline and the frozen-representation comparison measure exactly how much legal pre-training adds once labels exist. A gap of a few points would be reported as such, not inflated.

Why not just use hedge lexicons or boilerplate similarity? Corporate hedging is formulaic, so a lexicon may catch most of it. The hedging that matters, though, is often structural: a carve-out three sentences later, a definition that narrows scope, or an unverifiable quantity that looks specific. The span annotations let us measure what fraction of downgrades come from lexicon-detectable cues. If that fraction is near total, the cheap method wins and the proposal's central claim is wrong for this domain.

Decisive Checks

The project's main hypothesis fails cleanly if SaulLM-54B does not beat Mixtral 8x7B on minimal-pair directional accuracy under the frozen-representation protocol. That protocol removes the instruction-following confound, so a failure there means the extra legal tokens did not buy sensitivity to legal force. If SaulLM wins on legalese passages but the gap vanishes on plain-English rewrites, the benefit is vocabulary familiarity rather than reasoning about obligation, which is a smaller but still useful finding.

On the substantive side, the coalition result is decisive in either direction. If the 74.4% versus 40.7% presence gap shrinks sharply under strength weighting, coalition signaling is largely coverage. If it persists or widens, membership tracks real commitment. We validate this re-scoring on a held-out set of companies scored fully by annotators, and require the automated profile to rank those companies the same way the annotators do before any claim about the full audit is made.