Language, Not Code Abstraction: Natural-language Training Shapes Brain-aligned Geometry

Iñigo ParraarXiv 2026 · arXiv

TL;DR

Does training on the abstract logic of computer code make a language model "brain-like"? This paper finds that the answer is largely no. While code-trained models achieve positive brain alignment scores, these scores are mostly a byproduct of the transformer architecture itself; only training on natural language (English) produces the specific internal representational geometry that truly mirrors human neural activity.

What Problem Does This Paper Address?

Large Language Models (LLMs) are surprisingly good at predicting human brain activity during language processing. A lingering question in the field is why. Is it because they learn the nuances of human language, or is it because they learn "abstract sequential and compositional structure"?

Because programming languages are highly structured and logical, some researchers hypothesized that code-trained models might align with the brain just as well as language-trained ones. If true, this would suggest that "intelligence" or "abstraction" is the driver of alignment, rather than language specifically. This paper tests that hypothesis by comparing models trained on everything from high-level Python to raw machine code against models trained on simple English stories.

Methods

The author trained 25 GPT-2-small models from scratch, using a shared tokenizer and training recipe across five different corpora. To ensure a fair comparison, they used:

  • Encoding Analysis: Fitting a ridge regression to predict fMRI/ECoG responses from model activations.
  • Representational Geometry: Using techniques like Centered Kernel Alignment (CKA) to see how reproducible internal states are across different training runs, and Representational Similarity Analysis (RSA) to compare the "dissimilarity matrices" of the model vs. the brain without fitting any parameters.
  • Ablations and Interventions: "Transplanting" blocks from language models into code models and retraining only specific parts (like embeddings) to see which components carry the "brain-like" signal.
The experimental design: five corpora, one shared pipeline, and multiple layers of analysis.

Datasets

  • Training Corpora:
    • TinyStories: Simple, synthetic English prose.
  • Python: High-level source code.
  • C functions, x86-64 assembly, and Machine code (hex): Three versions of the same underlying programs at different levels of abstraction.
  • Brain Benchmarks:
    • Pereira-243/384: fMRI responses to sentences.
  • Tuckute2024: Language-network fMRI responses.
  • Blank2014: fMRI responses to stories.
  • Fedorenko2016: ECoG (electrodes on the brain) responses to words in sentences.

Key Insights and Ideas

  • The "Random" Baseline is High: A key finding is that a randomly initialized transformer already gets a "positive" brain score. This is due to the architecture (attention, positional encoding) and the tokenizer. Therefore, a model's success should be measured by how much it improves over a random baseline, not its absolute score.
  • No "Abstraction Ladder": The researchers expected a hierarchy where Python > C > Assembly > Binary in terms of brain alignment. Instead, they found no such ordering. Code models generally performed similarly to each other and stayed near the random-weight level.
  • Geometry vs. Accuracy: A model can be "reproducible" (getting the same internal states across different seeds) without being "brain-like." For example, binary-trained models were highly reproducible but occupied only about three effective dimensions, whereas the English-trained models expanded to about 26 dimensions.
  • The Embedding Bottleneck: Interestingly, updating just the embedding matrix of a code model with English data improved its brain alignment more than further training on its native code. This suggests the "interface" between the tokens and the transformer blocks is a major factor in alignment.

Key Results

On the primary benchmarks (Pereira-243 and Tuckute2024), the English-trained TinyStories models significantly outperformed all code models.

  • Encoding Advantage: TinyStories exceeded the code mean by $\Delta r = 0.053$ and $0.059$. As seen in Figure 2, this advantage persists across almost the entire depth of the model.
  • Fit-Free RSA: In tests that don't allow for a "fitted" readout, TinyStories' correspondence with neural data peaked at $\rho = 0.14$, which is double the level of code and random-weight models.
  • Benchmark Specificity: The "language advantage" wasn't universal; it disappeared on the Blank2014 and Fedorenko2016 benchmarks, highlighting that "brain-likeness" depends heavily on the specific dataset and neural recording method used.
Encoding scores across blocks. TinyStories (orange) shows a clear, depth-wide advantage over code models and random weights.
Table 1: Mean-over-depth scores. Note the bolded TS-code column showing the significant advantage of natural language training.

Limitations and Future Work

  • Architecture: The study was limited to GPT-2-small. Larger models or different architectures (like Mamba or RWKV) might show different dynamics.
  • Corpus Diversity: TinyStories is a very specific, simplified version of English. Future work could explore whether more complex or diverse natural language data further widens the gap.
  • The "Stitch" Problem: The block transplant experiments were done without a "learned stitch" (a transformation layer between blocks), which might have masked some potential for code-to-language transfer.
  • Domain Specificity: The study compares models against the human language network. It does not address how models might align with the brain's "Multiple Demand" network, which is known to be active during code comprehension in humans.

Summary updated

Abstract

Code-trained language models produce positive brain scores, which raises the question of whether abstract structure learned from programming languages can substitute for natural-language training. We trained 25 GPT-2-small models on TinyStories, Python, C functions, x86-64 assembly, and the corresponding machine code, and scored them using a shared tokenizer and encoding procedure against matched random-weight models. On Pereira-243 and Tuckute2024, TinyStories exceeds the mean of the four code models by $Δr = 0.053$ and $0.059$ (Holm-corrected $p<0.001$), while code training leaves alignment near the random-weight level. The language advantage is benchmark-specific: Blank2014 shows no corrected language-code difference, and Fedorenko2016 shows no language benefit. Geometry tracks the encoding result on Pereira-243: TinyStories seeds converge on the most similar representations, and its fit-free correspondence with neural dissimilarity peaks at $ρ= 0.14$, twice the code and random-weight levels. Binary-trained models have about three effective dimensions, so seed agreement alone does not imply neural correspondence. Updating only the embedding matrix on English improves mean-over-depth alignment for every code model more than an equal update on its own corpus. Within this architecture and training setup, natural-language training produces a distinct representational geometry with closer neural correspondence on Pereira-243
BibTeX
@misc{parra2026language,
  author = {Iñigo Parra},
  title = {{Language, Not Code Abstraction: Natural-language Training Shapes Brain-aligned Geometry}},
  year = {2026},
  eprint = {2611.09027},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09027},
}
Figure