TL;DR
The authors introduce BOUNDED-RS, a rigorous evaluation protocol that determines if a neural network’s systematic behavior (like understanding "Mary loves John" after seeing "John loves Mary") is actually driven by specific internal representations. By combining behavioral tests, decoding probes, and causal interventions with strict "validity gates," the protocol provides falsifiable verdicts on whether a proposed internal mechanism is truly being used or is merely a decodable byproduct.
What Problem Does This Paper Address?
Neural networks often show "behavioral systematicity"—they can handle novel combinations of familiar parts. However, the field has long debated representational systematicity: does the model actually represent those parts and their roles (e.g., "agent" vs. "theme") to produce the output?
Current interpretability tools are often used in isolation, leading to misleading conclusions:
- Probes can decode a signal that the model never actually uses.
- Interventions (like activation patching) can push a model into "off-manifold" states, causing changes that aren't representative of the model's natural logic.
- Failed searches for a representation are often interpreted as "the model doesn't have it," when the search might simply have been too narrow.
BOUNDED-RS aims to turn these "implicit rules" into a formal, calibrated protocol that adjudicates between four states: Unsupported, Present-only, Causal-use, or Undetermined.
Methods
The BOUNDED-RS protocol is structured into three distinct layers, as illustrated in Figure 1:
- Data Layer: Uses a "matched-set" testbed. For every "hard target" (e.g., a passive sentence), there is an "easy source" (active sentence) with the same meaning, plus variants that swap roles or fillers. This allows for exact counterfactual comparisons.
- Evidence Layer: Collects four types of evidence:
- Behavioral Anchor: Does the model actually succeed on the task?
- Recoverability: Can the hypothesized signal be decoded from hidden states?
- Intervention Tests: Does manipulating the signal (interchange, necessity, or repair) change the output more than a control would?
- Validity Checks: Are the intervention effects "realistic" (on-manifold) and "item-specific" (linked to the specific change made)?
- Decision Layer: Maps this evidence to a final verdict using control-calibrated scores and "stop rules."
A key innovation is the concept of Closure. To claim a family of explanations is Unsupported, a researcher must define a "closure basis" (a finite search space of candidate objects). If every object in that space fails the causal-use tests, the negative verdict is "bounded" by that space.
Datasets
- Semantic-Parsing Testbed: A new dataset of 96 matched sets (672 items total). It includes active/passive voices and relative clauses.
- SLOG Subdomain: Used for transfer testing, this is a structural generalization benchmark for semantic parsing.
- Audited Models: The authors primarily audited fine-tuned T5-small and T5-base models.
Key Insights and Ideas
- "Recoverability is not use": The paper demonstrates that decodability is a poor proxy for causal necessity. In their tests, taking recoverability as evidence of use led to 13 false "Causal-use" claims out of 34 noncausal cases.
- Adjudication over individual metrics: By requiring interventions to pass "validity gates" (realism and item linkage), the protocol filters out spurious effects that look like causal use but are actually just the model reacting to corrupted internal states.
- Two Explanatory Families: The authors contrast two popular theories of systematicity:
- BINDING: Role-filler signals (e.g., "who is doing what").
- GEOMETRY: Distributed structures like relation subspaces or cross-step bridges.
Key Results
- BINDING is Causal-use: For T5 models, the role-filler binding explanation was consistently rated as Causal-use. The model represents these roles and uses them to generate the correct parse.
- GEOMETRY is Unsupported: Within the declared search space (the closure basis), distributed geometric structures did not pass the causal-use gates.
- High Reliability: On 41 cases with known causal status, BOUNDED-RS licensed a false Causal-use verdict in only 1 out of 34 noncausal cases. In contrast, using ungated interventions would have resulted in 7 false positives.
- Robustness: These verdicts held across 487 different parameter configurations, multiple training seeds, and larger model sizes (T5-base).
Limitations and Future Work
- Architecture Scope: The protocol was only tested on T5 (encoder-decoder) models. Its applicability to decoder-only Large Language Models (LLMs) remains to be seen.
- Synthetic Data: The testbed is template-based and synthetic. While this allows for "clean" interventions, it doesn't capture the messiness of natural language.
- Tool Inheritance: BOUNDED-RS is a meta-protocol; its accuracy is still limited by the quality of the underlying probes and intervention methods it orchestrates.
- Manual Effort: Each new "explanatory family" requires researchers to manually define and validate at least six mechanism-specific interfaces.
Summary updated