Some question-answering models read a question with a language model while a graph neural network (a model that passes messages between connected nodes) reasons over a small knowledge-graph subgraph retrieved for that question. The two halves talk to each other at every layer through a shared "interaction" token or node. These models beat text-only baselines on commonsense and medical exams, and their authors explain the gain with a two-way story. The text is supposed to tell the graph which paths matter, for example by suppressing a distractor when the question says "not." The graph is supposed to give the text explicit relational facts. Nobody has checked either half of this story inside the network. All the evidence is end-to-end accuracy after fine-tuning.
We propose to measure each direction separately with linear probes. A linear probe is a single linear classifier trained on frozen internal activations; if it succeeds, the information is laid out in an easily readable way. The key is a matched control for each direction. To test text-to-graph flow, we ask whether the graph-side node states encode facts that only the text knows, such as negation or which answer the question favors. We then subtract what a graph encoder with no access to the question can encode. To test graph-to-text flow, we ask whether the text-side states encode facts that only the subgraph knows, such as path length or relation type. We then subtract what the same language model encodes from text alone. Finally, we erase the probed directions and check whether accuracy drops on the questions the story says should depend on them. The result is a per-layer, per-direction map of what fusion actually moves, and a causal check of whether the model uses it.
Core Claim
Bidirectional fusion is two hypotheses, and they may not hold equally. We expect graph-to-text flow (graph facts showing up in the language states) to be strong and easy to decode. We expect text-to-graph flow (question-dependent pruning, especially under negation) to be weaker, concentrated in the interaction node, and only partly used by the model. If that pattern holds, the "semantic pruning" explanation for the negation gains is overstated, and those gains mostly come from the language side absorbing graph facts.
We define directional flow at layer $\ell$ as
$$\Delta_{A\to B}(\ell) = \mathrm{Acc}\big(p_\ell;\ h^{B}_{\ell},\ y_A\big) - \mathrm{Acc}\big(p_\ell;\ \tilde h^{B}_{\ell},\ y_A\big),$$
where $y_A$ is a label computable only from modality $A$. The symbol $h^{B}_{\ell}$ is the frozen state of the fused model on modality $B$'s side at layer $\ell$. The symbol $\tilde h^{B}_{\ell}$ is the matching state of a unimodal control that never sees modality $A$. The term $p_\ell$ is a linear probe fit with the same budget in both cases. Both accuracies are reported net of a shuffled-label control task, so that probe memorization is not mistaken for encoding.
Research Program
The models are QA-GNN, GreaseLM and DRAGON. QA-GNN puts the whole question into the graph as a context node and scores knowledge-graph nodes for relevance to the question. GreaseLM interleaves language-model and graph layers that exchange information through an [INT] token and an interaction node. DRAGON pretrains this kind of fused encoder with masked-word prediction plus link prediction, a self-supervised objective that predicts missing graph edges.
These three span a useful contrast. QA-GNN and GreaseLM learn fusion only during task fine-tuning. DRAGON learns it in pretraining. Probing DRAGON's frozen pretrained checkpoint therefore tests whether its claimed reasoning ability exists before any task-specific update. That cannot be read off its fine-tuned accuracy. We will use released checkpoints and code where available and retrain otherwise, with ConceptNet subgraphs for CommonsenseQA and OpenBookQA and UMLS for MedQA.
We probe three sites at every interleaved layer: the [INT]/context token, the pooled text tokens, and individual graph-node states.
Text-only labels, used for text-to-graph flow, are of three kinds:
- whether the question contains a negation;
- whether a given node is the gold answer entity;
- whether a node lies on a path to the gold answer, given the question.
These labels change with the question while the subgraph stays fixed. That property is what makes them a test of conditioning.
Graph-only labels, used for graph-to-text flow, are three properties of the retrieved subgraph:
- the shortest-path length between question and answer entities;
- the relation type of the first edge on that path;
- whether the answer entity was retrieved at all.
The last label doubles as a retrieval-coverage diagnostic, tied to the retrieval bottleneck these systems share.
To isolate conditioning from co-occurrence, we build minimal pairs from CommonsenseQA and OpenBookQA items. Each pair keeps the same retrieved subgraph and the same answer set but inserts or removes a negation, so that the correct answer changes. Only a question-conditioned graph state can separate the two members of a pair. The pairs are validated by human annotators. This borrows the counterfactual pairwise-probing idea from Probing Graph Representations, which measured latent shifts between molecules differing in one property. Here the counterfactual is in the text and the measurement is on the graph.
The causal step uses linear erasure. For each layer and direction we project out the probe's learned subspace at the interaction site during the forward pass. We then measure the change in question-answering accuracy separately on negated and non-negated items. The comparison is against erasing random subspaces of the same rank. This responds directly to Probing the Probing Paradigm, which warns that probe accuracy need not imply task relevance.
Relation to Prior Work
The closest work is Probing Cross-Modal Representations in Multi-Step Relational Reasoning. It probed the vision–language model LXMERT with linear, support-vector-machine (SVM) and small neural-network (MLP) probes, finding that context-dependent size thresholds are linearly decodable and that an SVM recovers scene configuration at 85.70%. That work supplies our method and shows that linear probes can work on relational reasoning. It does not separate the two directions of exchange, uses no unimodal controls, has no causal step, and studies vision rather than knowledge graphs.
Probing Graph Representations constrains our design. It found that randomly initialized graph models already encode a lot of structure, and that structural information peaks in middle layers. Both findings force us to include random-initialization controls and to probe every layer. It studied only graph-only models.
ERAlign found that aligning a graph network and a large language model at intermediate layers beats aligning only their outputs. That motivates our prediction that flow is layer-specific, but ERAlign never measured flow directly.
Baselines
Do nothing. The naive reference is a majority-class probe plus a shuffled-label control task for every target. These calibrate what "decodable" means for each label.
Random initialization. The first architectural control is each fusion model with random weights, probed identically. Following the random-initialization finding in Probing Graph Representations, this separates what the wiring alone provides from what was learned. Pretraining has to beat this, not zero.
Unimodal controls. These are the $\tilde h$ terms in the flow measure:
- RoBERTa-large on the question text alone, for graph-to-text flow;
- the same graph network run on the identical subgraph with question-agnostic node initialization, for text-to-graph flow.
Both are trained on the same data, with the same probe regularization search, and the same number of probe fits per layer.
Strongest competitor. The strongest competitor is a text-only model given the retrieved paths written out as sentences, then probed linearly. It needs no graph network, so if it matches the fused model's graph-to-text decodability, it answers that half of the question more cheaply. It gets the same subgraphs and the same tuning budget. For context on task accuracy, the reported numbers on the CommonsenseQA in-house split are 74.2% for GreaseLM, 73.4% for QA-GNN and 68.7% for RoBERTa (GreaseLM). DRAGON reports a +5% average absolute gain over these across nine tasks (DRAGON). We found no reported result on Lacuna for linear probes on any of these models, nor for the frozen DRAGON checkpoint on CommonsenseQA.
What counts as a win. The fused model must show a positive $\Delta$ that clearly exceeds the shuffled-control gap. It must also beat the verbalized-path text model on graph-only labels, or on the minimal-pair negation labels.
Key ablation. The ablation that isolates our main ingredient is random-subspace erasure versus probe-subspace erasure. It shows whether decodable directions are causally used.
Why not just…?
Why not just report accuracy on the negation subset, as QA-GNN did? QA-GNN reports a 4.6% gain on negation questions. A gain there is consistent with either direction doing the work, and with neither, since the language side could handle negation better because of fine-tuning alone. Only directional probes with unimodal controls, plus erasure, can say which pathway carries the effect.
Why not just put the retrieved paths in the text and probe a plain language model? This is the real threat. Commonsense facts are rarely stated in raw text: BiDAF++ with 100 web snippets per question reached only 32.0% on CommonsenseQA. That argues for supplying structure, but it does not show that a graph network is needed once the structure is written out. We run this baseline at equal budget. If it matches the fused model's graph-to-text flow, we will report that graph-to-text fusion adds nothing representationally. The study would then narrow to text-to-graph conditioning, which this baseline cannot test because it has no graph states.
Why not just use MLP probes or fine-tuning, which can read more complex information? Expressive probes can build the target feature themselves, and fine-tuning changes the weights we want to audit. We still run one-hidden-layer MLP probes as a capacity ladder. A large gap between linear and MLP accuracy, measured against the shuffled-label controls, is itself a finding: the information is present but not linearly organized. That bears on the concern that multi-hop relations may not be linearly readable at all.
Proposed Experiments
The main experiment probes all three models at every interleaved layer on CommonsenseQA and OpenBookQA, with MedQA/UMLS as a domain-transfer check. We compute $\Delta_{T\to G}$ and $\Delta_{G\to T}$ separately for the interaction site and for ordinary nodes or tokens.
Decisive check one is the minimal-pair test. Graph-node states in GreaseLM or DRAGON must separate negated from non-negated twins above the question-agnostic graph control. If they do not, the semantic-pruning explanation fails.
Decisive check two is erasure. Removing the text-to-graph subspace at the interaction node should lower accuracy on negated items more than on non-negated ones, and more than random erasure of the same rank. If accuracy holds, the direction is decodable but unused.
Decisive check three compares DRAGON's frozen pretrained checkpoint with its fine-tuned version. If flow appears only after fine-tuning, then DRAGON's claimed advantage of a pretrained reasoning ability is mostly adaptation.
Finally, we will report a layer profile for each direction. This tests whether fused models keep graph structure up to the output layer or discard it in late layers, as the graph-only networks in Probing Graph Representations do.
References
- QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
- GreaseLM: Graph REASoning Enhanced Language Models
- Deep Bidirectional Language-Knowledge Graph Pretraining
- Probing Graph Representations
- Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
- Probing Cross-Modal Representations in Multi-Step Relational Reasoning
- ERAlign: Energy-based Representation Alignment of GNNs and LLMs on Text-attributed Graphs
- CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge