A Bounded Evaluation Protocol for Representational Systematicity

Boyu Cao, Wenshuo WangarXiv 2026 · arXiv

TL;DR

The authors introduce BOUNDED-RS, a rigorous evaluation protocol that determines if a neural network’s systematic behavior (like understanding "Mary loves John" after seeing "John loves Mary") is actually driven by specific internal representations. By combining behavioral tests, decoding probes, and causal interventions with strict "validity gates," the protocol provides falsifiable verdicts on whether a proposed internal mechanism is truly being used or is merely a decodable byproduct.

What Problem Does This Paper Address?

Neural networks often show "behavioral systematicity"—they can handle novel combinations of familiar parts. However, the field has long debated representational systematicity: does the model actually represent those parts and their roles (e.g., "agent" vs. "theme") to produce the output?

Current interpretability tools are often used in isolation, leading to misleading conclusions:

  • Probes can decode a signal that the model never actually uses.
  • Interventions (like activation patching) can push a model into "off-manifold" states, causing changes that aren't representative of the model's natural logic.
  • Failed searches for a representation are often interpreted as "the model doesn't have it," when the search might simply have been too narrow.

BOUNDED-RS aims to turn these "implicit rules" into a formal, calibrated protocol that adjudicates between four states: Unsupported, Present-only, Causal-use, or Undetermined.

Methods

The BOUNDED-RS protocol is structured into three distinct layers, as illustrated in Figure 1:

  1. Data Layer: Uses a "matched-set" testbed. For every "hard target" (e.g., a passive sentence), there is an "easy source" (active sentence) with the same meaning, plus variants that swap roles or fillers. This allows for exact counterfactual comparisons.
  2. Evidence Layer: Collects four types of evidence:
  3. Behavioral Anchor: Does the model actually succeed on the task?
  4. Recoverability: Can the hypothesized signal be decoded from hidden states?
  5. Intervention Tests: Does manipulating the signal (interchange, necessity, or repair) change the output more than a control would?
  6. Validity Checks: Are the intervention effects "realistic" (on-manifold) and "item-specific" (linked to the specific change made)?
  7. Decision Layer: Maps this evidence to a final verdict using control-calibrated scores and "stop rules."
Figure 1: Three-layer structure of BOUNDED-RS. The matched-set testbed defines controlled contrasts; family-specific evidence operations test recoverability and causal use; realism, linkage, and adjudication rules translate the evidence into bounded state decisions.

A key innovation is the concept of Closure. To claim a family of explanations is Unsupported, a researcher must define a "closure basis" (a finite search space of candidate objects). If every object in that space fails the causal-use tests, the negative verdict is "bounded" by that space.

Datasets

  • Semantic-Parsing Testbed: A new dataset of 96 matched sets (672 items total). It includes active/passive voices and relative clauses.
  • SLOG Subdomain: Used for transfer testing, this is a structural generalization benchmark for semantic parsing.
  • Audited Models: The authors primarily audited fine-tuned T5-small and T5-base models.

Key Insights and Ideas

  • "Recoverability is not use": The paper demonstrates that decodability is a poor proxy for causal necessity. In their tests, taking recoverability as evidence of use led to 13 false "Causal-use" claims out of 34 noncausal cases.
  • Adjudication over individual metrics: By requiring interventions to pass "validity gates" (realism and item linkage), the protocol filters out spurious effects that look like causal use but are actually just the model reacting to corrupted internal states.
  • Two Explanatory Families: The authors contrast two popular theories of systematicity:
  • BINDING: Role-filler signals (e.g., "who is doing what").
  • GEOMETRY: Distributed structures like relation subspaces or cross-step bridges.
Figure 2: Two-family audit workflow. Layer trajectories and routes are GEOMETRY diagnostics; the closure basis consists of interaction objects, cross-step bridges, and query–source relations. A family-final GEOMETRY negative requires closure, and an ontology-level negative would further require validated detectors.

Key Results

  • BINDING is Causal-use: For T5 models, the role-filler binding explanation was consistently rated as Causal-use. The model represents these roles and uses them to generate the correct parse.
  • GEOMETRY is Unsupported: Within the declared search space (the closure basis), distributed geometric structures did not pass the causal-use gates.
  • High Reliability: On 41 cases with known causal status, BOUNDED-RS licensed a false Causal-use verdict in only 1 out of 34 noncausal cases. In contrast, using ungated interventions would have resulted in 7 false positives.
  • Robustness: These verdicts held across 487 different parameter configurations, multiple training seeds, and larger model sizes (T5-base).

Limitations and Future Work

  • Architecture Scope: The protocol was only tested on T5 (encoder-decoder) models. Its applicability to decoder-only Large Language Models (LLMs) remains to be seen.
  • Synthetic Data: The testbed is template-based and synthetic. While this allows for "clean" interventions, it doesn't capture the messiness of natural language.
  • Tool Inheritance: BOUNDED-RS is a meta-protocol; its accuracy is still limited by the quality of the underlying probes and intervention methods it orchestrates.
  • Manual Effort: Each new "explanatory family" requires researchers to manually define and validate at least six mechanism-specific interfaces.

Summary updated

Abstract

Multiple studies have shown that neural networks can be behaviorally systematic, producing correct outputs for novel combinations of familiar parts. Whether they are also representationally systematic, that is, whether they represent these parts and their roles and use them to produce the outputs, remains hard to test, because each kind of evidence can mislead on its own: correct outputs can come from shortcuts, a decodable signal need not be used, an intervention can act through unintended pathways, and a failed search does not show absence. We propose a bounded evaluation protocol for representational systematicity (Bounded-RS): for an explanation that attributes a systematic behavior to a declared representation, it tests on a matched behavioral contrast whether that representation is recoverable and whether it is causally used, checks the validity of this evidence, and assigns the explanation one of four bounded states, Unsupported, Present-only, Causal-use, or Undetermined, through control-calibrated scores, gates, and stop rules. On a semantic-parsing testbed of 96 matched sets that we release, the tested role-filler Binding explanation of fine-tuned T5 models is Causal-use and the distributed Geometry explanation is Unsupported within its declared search space, verdicts that hold across calibration-compatible parameter configurations, training seeds, and T5-base. On 41 cases with known causal status, full adjudication licenses a false Causal-use in 1 of 34 noncausal cases, against 13 when recoverability is taken as use, without missing any causal case. Bounded-RS thus turns the question of representational systematicity, debated since Fodor and Pylyshyn, into comparable and falsifiable verdicts on declared explanations, each reported with the evidence pattern that licenses it and, for a negative, the search space that bounds it.
BibTeX
@misc{cao2026bounded,
  author = {Boyu Cao and Wenshuo Wang},
  title = {{A Bounded Evaluation Protocol for Representational Systematicity}},
  year = {2026},
  eprint = {2611.01755},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01755},
}
Figure