When people reason that "A is taller than B, B is taller than C, so A is taller than C," they appear to place the items on an internal line. One behavioral trace of this is the symbolic distance effect: comparing items far apart in the order is easier than comparing neighbors. Neuroscience links this kind of inference to "cognitive maps," low-dimensional geometric codes in the hippocampus and entorhinal cortex that lay out abstract relations the way they lay out physical space. Separately, interpretability work on multilingual language models finds that the middle layers act as a shared "semantic hub," where sentences in different languages converge to similar representations before the model turns them back into language-specific words. This proposal joins the two findings. We ask whether a multilingual model, given ordering premises in context in any language, builds one ordered map of the items in its middle layers, and whether the axis of that map is the same across languages. We also ask whether a missing map explains why relational reasoning fails in some languages.
The question matters because multilingual failures are currently diagnosed with blunt tools: whole-hidden-state similarity to English, or patching entire activations. If a specific, low-dimensional relational geometry turns out to be the shared currency of the hub, failures can be separated into two kinds. In one, the model never builds the map. In the other, it builds the map but cannot read the entities. Each kind calls for a different fix, and a narrow, geometry-targeted intervention would serve as a causal test of the hub hypothesis that is stronger than "representations look similar."
Core Claim
We claim that in-context transitive inference in a multilingual decoder is carried by a roughly one-dimensional rank axis in the middle layers. We further claim that this axis is shared across languages, so a direction fitted on English premises decodes, and causally controls, item rank when the premises are in Telugu, Swahili, or Thai. Where accuracy on non-adjacent pairs collapses in a language, we predict the rank axis is absent or noisy for that language, and the symbolic distance effect flattens. A flat distance effect is the signature Relational reasoning and inductive bias in transformers trained on a transitive inference task associates with "lazy" in-context copying of adjacent premises rather than true inference.
What is new is the combination. We use a metric, behaviorally anchored geometry from cognitive mapping, and we localize it in depth and test it across languages. Prior multilingual work tests only generic alignment, and prior relational-geometry work is monolingual.
Relation to Prior Work
Two papers supply the relational side and motivate the gap. Relational reasoning and inductive bias in transformers trained on a transitive inference task shows that small transformers trained in-weights form a linear rank manifold with a distance effect. In-context learning in the same models falls to chance on non-adjacent pairs, and a "number line" prompt helps large models. That work is English-only and never asks where in depth the manifold forms. Investigating Representation Universality: Case Study on Genealogical Representations finds causally relevant cone-shaped subspaces for family trees in several LLMs. It reports universality peaking in early-to-mid layers and performance degrading when tree facts are presented out of order. It too is monolingual. We reuse its probe-plus-patching methodology for a simpler geometry, a line, and add the cross-lingual dimension.
On the multilingual side, The Semantic Hub Hypothesis and LinguaMap establish the depth pipeline: language-specific early layers, a shared middle, and language control late. Can you map it to English? shows that patching English activations into layers 14–25 flips failing non-English answers to correct ones. These papers motivate where to look, and they supply the patching technique. All three treat the hub as an undifferentiated vector space, whereas we ask what geometry it carries for one well-defined relational computation.
One result constrains the proposal. Large Language Models Are Cross-Lingual Knowledge-Free Reasoners reports a cross-lingual transfer ratio above 90% for 7 of 9 languages on knowledge-free logic and arithmetic, with alignment in middle layers. That result could imply there is no transitive-inference failure to explain. We take this seriously. Its tasks do not isolate non-adjacent in-context inference, which is exactly where Geerts et al. find English itself fails. If transfer turns out uniformly high, the probing results still answer whether the map is shared, but the failure-diagnosis claim would be falsified.
The neuroscience literature sets the geometric hypotheses but is not imported as mechanism. The Hippocampus Maps Concept Space, Not Feature Space shows hippocampal distances track task-relevant conceptual dimensions rather than surface features. The analog here is that rank geometry should be invariant to surface language. A Cognitive Model for Learning Abstract Relational Structures from Memory-based Decision-Making Tasks argues that generalization requires separating relational structure from entity identity, and reports that standard transformers fail at this. Our cross-lingual test is a natural-language version of that separation. Entity tokens change completely across scripts, and the question is whether the structure survives.
Research Program
We first build a multilingual transitive-inference suite in the style of ReCogLab, as used by Geerts et al. Each problem contains orderings of 7 items, matching the list length in Hosoya's 1D task, and 12 items, to stress capacity. The items are nonce names or translated common nouns. Adjacent-pair premises are given in shuffled order, which addresses the order sensitivity Baek et al. report. Queries cover all pairs at each symbolic distance. Templates are rendered in the 10 KFRD languages, plus Telugu, Swahili, Thai, and Quechua, to span high- and low-resource languages and Latin and non-Latin scripts. Native-speaker checks are used where available. Because the task needs no world knowledge, accuracy differences reflect relational processing and entity reading, not missing facts.
Next, for open-weight models (Llama-3-8B and a Qwen model), we fit per-layer linear probes from each item's final-token hidden state to its rank, separately for each language. We record three measures. The first is within-language rank decodability, measured as held-out Spearman correlation. The second is cross-language transfer: a probe trained on English and applied to language L. The third is the angle between per-language rank directions. A shared map predicts that cross-language transfer peaks in the same middle band where Ravisankar et al. found effective patching, and that it stays low in early layers. This band structure also addresses the tension with Baek et al.'s early-layer universality, because it tests whether relational geometry and cross-lingual alignment peak at the same depth.
Third, we run causal edits. We shift one item's projection along the English-fitted rank direction toward another item's rank, and we measure whether answers to queries involving that item change in the direction the new rank predicts. We do this in English and in each target language. If an English-derived direction steers Telugu answers, the map is both shared and used.
Fourth, we test the failure diagnosis. For each language we relate non-adjacent accuracy and the slope of the distance effect to rank decodability, and we separately relate them to an entity-reading control. That control is a probe for whether the hidden state identifies which premise mentioned the item, which tests whether the model can tell the entities apart at all. The two-cause account predicts two clusters of failing languages. One has good entity reading and no rank axis, meaning the map was never built. The other has poor entity reading, likely from token fragmentation in non-Latin scripts.
Finally, we run a discriminating training experiment. We train a middle-layer LoRA adapter (a small low-rank weight update) on English-only synthetic transitive inference, with an auxiliary loss that makes rank linearly decodable along a single direction. We compare it with the same adapter trained with the middle-layer cross-lingual contrastive loss of Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs. If map-formation failures dominate, the English-only map loss should transfer zero-shot to map-failing languages. If entity-reading failures dominate, contrastive alignment should help more. The outcome tells us which bottleneck is real.
Baselines
The do-nothing reference is the base model prompted directly in the target language. Its per-distance accuracy curve defines the failure we explain. We have no reported Lacuna result for multilingual in-context transitive inference, so these numbers will be produced fresh.
The upper reference is translate-test: premises machine-translated to English, then answered in English. Belebele and XCOPA both report that this often beats direct target-language prompting. It is run on the same items with the same model. It is not a competitor for the mechanistic claim, but it bounds how much of the gap is language-specific.
The strongest prompting baseline is the number-line prompt of Geerts et al., given in English instructions and in translated instructions. They report that it significantly improves hard cases in Gemini and Gemma but give no numeric value usable here. Beating it means the rank-direction intervention or adapter improves non-adjacent accuracy in failing languages beyond what the prompt achieves, at equal or lower cost.
The strongest generic internal method is INCLINE from Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention. It applies least-squares alignment of last-token states toward English and reports +4.96 accuracy on XStoryCloze and +3.50 on MGSM. We fit it on the same parallel data used for our probes. Beating it on transitive inference with a one-dimensional edit, rather than a full alignment matrix, would show the relevant shared information is narrow.
The key ablation compares three patches with matched norm: rank-direction-only, full middle-layer hidden-state patching as in Ravisankar et al., and a random direction. This isolates whether the rank axis carries the effect.
For training, the English map-loss adapter is compared with the contrastive-alignment adapter, task-only LoRA, and full fine-tuning. All use the same data, steps, and hyperparameter search budget. We also check XCOPA and Belebele to confirm general multilingual ability is not harmed.
Why not just…?
Why not just translate the premises to English? For high-resource languages with synthetic premises this will likely match or beat everything here, and we say so plainly. The proposal does not claim a better product for those languages. Its aim is a mechanistic account of whether relational structure is language-neutral inside the model. Translate-test is still informative where machine translation is weak. LangBridge motivates this case: MGSM accuracy in underrepresented languages rose from 6.0% to 45.2% through internal mapping rather than translation. Quechua and Telugu in our suite test whether translate-test degrades while an English-fitted rank edit still works.
Why not just use a number-line prompt? It is cheap and documented to help. But Geerts et al. also show that a circle prompt hurts, and Baek et al. show that ordering matters, so prompting may shape behavior without producing a robust map. Our probes test this directly. If the number-line prompt restores both accuracy and a cross-lingually decodable rank axis in failing languages, the intervention adds nothing for control purposes, and the contribution reduces to the diagnosis.
Why not just align the whole middle layer to English, as INCLINE or full patching do? This is the real threat to the "geometry matters" framing. The matched-norm ablation settles it. If full-state patching beats the rank-direction edit by a wide margin, then the relevant shared content is not a single axis, and the cognitive-map framing is decorative.
Why not just fit a linear probe and skip the cognitive-map language? Decodability alone is weak evidence of a map. The map hypothesis makes metric predictions: probe projections should be monotone in rank, the distance effect should follow projection gaps, and edits along the axis should shift answers. A probe that decodes rank without these properties would count against the hypothesis.
Decisive Checks
The central claim survives if three things hold together. First, in at least the middle band of layers, English-fitted rank probes transfer to most target languages. Second, edits along that direction flip answers in target languages roughly as often as in English, and more often than random directions. Third, languages that fail on non-adjacent pairs while reading entities well show weak rank decodability.
The claim fails in either of two cases. Rank directions might be language-specific, with near-orthogonal per-language axes. Or rank decodability might be uniformly high while failures track only entity reading, which would mean the bottleneck is tokenization, not map formation. The adapter comparison then decides which fix matters more. Either outcome replaces the current vague picture of a "semantic hub" with a measured account of what one shared relational computation looks like inside it.
References
- Relational reasoning and inductive bias in transformers trained on a transitive inference task
- Investigating Representation Universality: Case Study on Genealogical Representations
- The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities
- LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?
- Can you map it to {E}nglish? The Role of Cross-Lingual Alignment in the Multilingual Performance of {LLM}s
- Large Language Models Are Cross-Lingual Knowledge-Free Reasoners
- The Hippocampus Maps Concept Space, Not Feature Space
- A Cognitive Model for Learning Abstract Relational Structures from Memory-based Decision-Making Tasks
- Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
- Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention
- LangBridge: Multilingual Reasoning Without Multilingual Supervision