Pixels over Symbols: Sensory Realism Improves Behavioral Alignment in Models of Cognition

Hao Yuan Bai, Xiaoxuan Lei, Zihan Weng, Motahareh Pourrahimi, Lucas Gomez, Pouya BashivanarXiv 2026 · arXiv

In the quest to build AI that "thinks" like we do, researchers have long relied on a convenient shorthand: symbols. Instead of showing a model a messy high-resolution image of a cat, we give it a clean vector—a "one-hot" code—representing "Cat." The assumption was that the logic of cognition is separate from the mechanics of seeing.

A new paper from researchers at McGill University and Mila, titled "Pixels over Symbols: Sensory Realism Improves Behavioral Alignment in Models of Cognition," challenges this fundamental assumption. They argue that if you want a model to behave like a human, you can't skip the pixels.

TL;DR

By comparing neural networks trained on abstract symbols versus raw images, this study demonstrates that sensory realism is a primary driver of human-like behavior. Surprisingly, the "thinking" part of the brain (recurrent weights) matters less for this alignment than the "seeing" part; even a model trained on symbols can act human-like if you simply plug in a realistic visual input.

What Problem Does This Paper Address?

For decades, computational neuroscience and cognitive modeling have treated sensory details as "incidental." The prevailing view was that once information reaches the higher-order brain regions responsible for memory and decision-making, it has already been abstracted into categorical symbols.

However, humans don't just make decisions; we make specific patterns of mistakes. We might confuse a dog for a cat more often than a dog for a car. Abstract models, where every symbol is mathematically equidistant from every other symbol, fail to capture these nuances. The authors set out to test whether "sensory realism"—training models on high-dimensional pixels rather than low-dimensional codes—is necessary for models to align with human behavioral patterns.

Methods

The researchers conducted a large-scale empirical comparison across three groups:

  1. Abstract Models: Sequence models (RNNs, LSTMs, Transformers) receiving symbolic one-hot vectors.
  2. Natural Models: The same architectures, but receiving image sequences processed through a frozen, pre-trained vision encoder (like CLIP or DINO).
  3. Vision-Language Models (VLMs): State-of-the-art models like Gemini 3.1 Pro, evaluated via natural language prompts.
Figure 1: Schematic of the Experimental Setup. Models receive both task and stimulus information in their inputs. Task indicator is provided in either textual or one-hot while stimuli are presented in their naturalistic or abstract formats. In the abstract format, information about the primary object in the scene is provided as a combination of one-hot (category, object identity, position) and scalar values. Three groups are studied: humans, neural network models trained on abstract or natural inputs, and pretrained vision-language models. Behavioral responses are collected from each group and used to estimate the choice probabilities for each task trial. These probabilities are then used to calculate behavioral consistency with humans using different metrics.

They evaluated these agents on a battery of 43 memory-dependent decision-making tasks (e.g., "Was the object I saw two frames ago the same category as this one?"). To measure "human-likeness," they used Human Consistency: a metric that correlates the trial-by-trial error patterns of models with those of 20 human participants per task.

Datasets

  • HvM (High-Variation Model): A dataset of 5,760 images featuring 8 object categories (cars, faces, etc.) placed on naturalistic backgrounds with varying rotations, scales, and positions.
  • iWISDM: A framework used to generate the 43 cognitive tasks, ranging from simple delayed match-to-sample to complex contextual decision-making.
  • Human Benchmarks: Behavioral data collected via Amazon Mechanical Turk for a subset of 12 tasks.

Key Insights and Ideas

  • The Geometry of Errors: Abstract symbols are "equidistant"—a model is just as likely to confuse a "Car" symbol with a "Face" symbol. Natural images, however, occupy a structured manifold. This "input geometry" naturally steers the model toward human-like error patterns.
  • Recurrent Weights are "Input-Agnostic": In a striking "substitution experiment," the authors took the recurrent weights of a model trained entirely on symbols and plugged them into a pipeline receiving natural images. The result? The model's behavior suddenly became human-like. This suggests that the internal "logic" learned from symbols is compatible with the "geometry" of pixels.
  • The VLM Paradox: While massive VLMs like Gemini 3.1 Pro often exceed human accuracy, they frequently fall behind task-trained "Natural" models in behavioral alignment. Being "smarter" doesn't necessarily mean being more "human."

Key Results

  • Superior Alignment: Natural models outperformed Abstract models in human consistency by a significant margin (~0.2 Pearson correlation).
  • Architecture vs. Encoder: The choice of vision encoder (e.g., using DINOv3 vs. CLIP) had a larger impact on human alignment than the choice of sequence architecture (e.g., LSTM vs. Transformer).
  • The Consistency Gap: Even the best Natural models only reached about 64% of the internal consistency observed between human groups, indicating that sensory realism is a necessary, but not sufficient, ingredient for perfect alignment.
Figure 2: Comparison of Abstract and Natural models across sequential architectures. (a) Correlation with human behavior. Gray line and zone indicates human internal behavioral con- sistency, ρ(h, h) in Eq. 1. Bars show ρ(m, h) averaged over 20 random split-halves in Fisher- z space; error bars show the standard deviation across split-halves. (b) Task accuracy. Mean and standard deviation reported as error bars, 6 seeds for models and 20 participants for hu- mans. VLMs (left to right): nemotron-nano-12b-v2, nemotron-nano-12b-v2-no-thinking, qwen3-vl-30b, qwen3-vl-235b, and gemini-3.1-pro-preview (surpasses human-level accu- racy).

Limitations and Future Work

The authors acknowledge several hurdles:

  • Sample Size: Human data was derived from a relatively small pool (20 per task).
  • Pre-training Confound: Natural models rely on encoders pre-trained on massive datasets (ImageNet). It’s hard to decouple the effect of "pixels" from the effect of "pre-training."
  • Task Simplicity: The tasks were binary (match/non-match). Future work should explore more open-ended cognitive behaviors.
  • Neural Dynamics: While behavioral alignment improved, the study doesn't yet prove that the internal neural trajectories of these models match the human brain. The authors suggest coupling behavioral metrics with brain-activity comparisons as the next logical step.

By proving that "pixels matter," this research suggests that neuroscientists should be wary of conclusions drawn from models that only play with symbols. If we want to understand the human mind, we have to look at the world through the same messy, high-dimensional lens that we do.

Summary updated

Abstract

Computational models of cognition have traditionally relied on abstract symbolic inputs that approximate sensory experiences with simplified vector representations. This simplification reflects historical computational constraints and a long-held assumption that the details of sensory realism are incidental to the study of cognitive processes. Recent advances have enabled the development of sensory-realistic models capable of performing diverse cognitive tasks, yet whether sensory realism meaningfully shapes cognition has not been formally established. We address this gap through a large-scale empirical comparison of abstract and sensory-realistic neural network models trained on a battery of memory-dependent decision-making tasks. Our results show that sensory-realistic models mirror human behavioral response patterns significantly more than their abstract counterparts, though a substantial gap to human internal behavioral consistency remains even for the best tested models. By analyzing the internal dynamics of these systems, we find that the learned recurrent weights are not the main source of this divergence. In fact, frozen abstract-trained weights driven by sensory-realistic inputs recover the human consistency of sensory-realistic models. The resulting dynamics are also within the seed-to-seed variability of sensory-realistic models. Together, our findings argue that across a broad set of visual working-memory tasks, sensory realism accounts for a substantial portion of the model-human behavioral agreement, more than the choice of sequence architecture and comparable to the choice of vision encoder. These results suggest that findings obtained from abstract task models in neuroscience should be more carefully interpreted in light of their potential behavioral limitations.
BibTeX
@misc{bai2026pixels,
  author = {Hao Yuan Bai and Xiaoxuan Lei and Zihan Weng and Motahareh Pourrahimi and Lucas Gomez and Pouya Bashivan},
  title = {{Pixels over Symbols: Sensory Realism Improves Behavioral Alignment in Models of Cognition}},
  year = {2026},
  eprint = {2611.10565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10565},
}
Figure