For years, a foundational assumption in computational neuroscience was that optimizing artificial neural networks (ANNs) for task performance—such as maximizing top-1 accuracy on ImageNet—would naturally drive them toward the computational solutions discovered by biological evolution. Early deep networks seemed to validate this, matching primate neural responses better than hand-engineered models.
However, recent work has uncovered a striking performance paradox: once models surpass a certain threshold of engineering success, further optimization actually causes them to diverge from biological systems. As Drew Linsley et al. demonstrate in Better artificial intelligence does not mean better models of biology, there is a critical turning point around 70% accuracy on ImageNet. Below this threshold, task accuracy and biological alignment improve together; above it, the relationship reverses, and higher accuracy correlates with poorer biological plausibility.
Measuring the Strategy Gap
Why do high-performing models stop acting like brains? The answer lies in how they solve tasks. A model can achieve identical aggregate accuracy to a human while using a completely alien strategy. To understand why the idea of a performance-plausibility trade-off works, we must look at how researchers quantify these strategic differences.
Error Consistency
To expose the strategy gap, Robert Geirhos et al. introduced the concept of "error consistency" in Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency. Instead of looking at overall percentages, they evaluated whether two systems fail on the exact same images. The authors showed that while high-performing convolutional neural networks (CNNs) are highly consistent with each other, they share almost no common error patterns with humans, proving that their underlying decision-making processes have diverged.
Decision Variable Correlation
Moving beyond behavior to internal representations, Yu Eric Qian et al. developed a metric called Decision Variable Correlation (DVC) in Quantifying Task-relevant Similarities in Representations Using Decision Variable Correlations. Their analysis confirmed that as models become more proficient at image classification, their internal mechanisms for processing task-relevant information diverge further from the primate visual cortex.
This phenomenon is not isolated to vision. In speech and audio processing, Federico Adolfi et al. (Successes and critical failures of neural networks in capturing human-like speech recognition) and Greta Tuckute et al. (Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions) have documented similar disconnects: models optimized purely for technical engineering benchmarks fail to replicate human auditory robustness and neural activity.
Why Optimization Drives Divergence
The root cause of this divergence is that standard deep learning models are unconstrained. They are free to exploit high-frequency statistical regularities, texture shortcuts, and global pixel relationships that are invisible or useless to biological systems. Biological brains, by contrast, operate under strict evolutionary, anatomical, and metabolic constraints.
The Cost of Foveation
Consider human vision: it is foveated. We see with high resolution only at the very center of our gaze, processing the periphery with significantly degraded clarity.
When Galen Pogoncheff et al. integrated biological constraints like cortical magnification and foveation into a standard architecture in Explaining V1 Properties with a Biologically Constrained Deep Learning Architecture, they observed a direct trade-off. The model's alignment with primary visual cortex (V1) data increased, but its standard benchmark accuracy decreased because it could no longer process off-center information efficiently.
Similarly, Danqing Shi et al. observed in Chartist: Task-driven Eye Movement Control for Chart Reading that restricting a model to human-like foveated vision patches improves the biological plausibility of its eye-movement paths but degrades overall question-answering accuracy compared to models that process the entire visual field simultaneously.
Representational Goals: Factorization vs. Compression
The primate brain's optimization goal is fundamentally different from a standard classifier's. A classifier is trained to discard non-identity information (like lighting or pose) to map an image to a label.
As Elias Issa et al. argue in Factorized visual representations in the primate visual system and deep neural networks, the primate visual system instead organizes information through factorization—representing scene parameters in independent, non-interfering neural subspaces. Standard task optimization hits a predictivity plateau because it does not incentivize this multi-variable preservation, causing performance-optimized models to evolve into worse models of the inferotemporal visual cortex (Drew Linsley et al., Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex).
Harmonizing Performance and Plausibility
If optimizing purely for accuracy drives models away from biology, how can we build models that serve as accurate scientific hypotheses for the brain—what Kohitij Kar et al. call "SMART models" in The Quest for an Integrated Set of Neural Mechanisms Underlying Object Recognition in Primates?
Brain-Driven Training
One approach is to change the optimization target by training models directly on neural data. Meenakshi Khosla et al. developed response-optimized encoding models in Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models, showing that brain-aligned training naturally recovers biological properties like shape bias.
However, a persistent discrepancy remains: these brain-optimized models often lack competitive classification accuracy on massive labeled datasets. As Zitong Lu et al. demonstrated in Teaching CORnet human fMRI representations for enhanced model-brain alignment, optimizing for biological alignment with human brain activity does not automatically translate to improved performance on standard computer vision benchmarks.
Pure Task Optimization ────► High Accuracy ──────► Low Biological Alignment
(Exploits statistical shortcuts)
Brain-Driven Training ────► High Alignment ─────► Lower Benchmark Accuracy
(Bound by biological constraints)
Harmonized Training ────► High Accuracy ──────► High Alignment
(Forced to use human-like features)
The Path Forward: Dual Objectives
To bridge this gap, researchers are developing hybrid training paradigms that enforce human-like constraints without sacrificing performance. In Harmonizing the object recognition strategies of deep neural networks with humans, Thomas FEL et al. introduced a "neural harmonizer"—a training module that forces models to align their internal feature attention with human visual priorities. This approach demonstrates that the trade-off is not an absolute law; by explicitly correcting the divergence during training, models can be guided to maintain high benchmark performance while utilizing biological strategies.
Ultimately, platforms like the Algonauts Project (Gemma Roig et al., The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence) provide the cross-disciplinary benchmarking necessary to navigate this tension, ensuring that as AI models scale, they remain faithful digital twins of biological intelligence rather than alien statistical engines.
Key Foundational Work
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (2019) — contributed 2 concept elements to this direction.
Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency (2020) — contributed 2 concept elements to this direction.
The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence (2019) — contributed 1 concept elements to this direction.