Abstract
Humans can distinguish visual stimuli that differ by features the size of only a few photoreceptors. This is possible despite the incessant image motion due to fixational eye movements, which can be many times larger than the features to be distinguished. To perform well, the brain must identify the retinal firing patterns induced by the stimulus while discounting similar patterns caused by spontaneous retinal activity. This paper describes a "Markov Decoder"—a biologically plausible algorithm that allows the brain to achieve sub-photoreceptor visual acuity by dynamically tracking these random eye movements through retinal spike patterns. By treating the stimulus position as a hidden variable and performing real-time inference, the brain effectively "de-jitters" the visual world.
Introduction: The Paradox of Visual Stability
Even when we attempt to stare fixedly at an object, our eyes are never truly still. They undergo continuous "fixational eye movements," including drift, tremor, and microsaccades. These movements shift the visual image across the retinal mosaic, often by distances much larger than the tiny details we are trying to perceive. This creates a significant computational challenge: how does the brain maintain a stable, high-resolution percept when the input is constantly jittering?
The brain faces two primary hurdles. First, there is no precise internal "readout" of these microscopic movements available to the visual system to cancel them out perfectly. Second, retinal ganglion cells exhibit spontaneous firing even in the absence of a stimulus. To perform accurately, the brain must distinguish stimulus-driven spikes from this background noise while the stimulus itself follows an unknown, jittery path across the retina.
Methods: The Markov Decoder
The authors developed a computational framework known as a Markov Decoder, based on a reaction-diffusion framework (specifically the Fokker-Planck equation). Instead of assuming a stable eye or a known trajectory, the decoder treats the stimulus position as a hidden variable following a random walk.
The algorithm operates in two alternating steps:
- Reaction Step: When a retinal spike occurs, the model updates its internal probability distribution, increasing the likelihood that the stimulus is located at a position consistent with that spike’s receptive field.
- Diffusion Step: In the intervals between spikes, the probability distribution "diffuses" or spreads out. This mathematically accounts for the increasing uncertainty about the stimulus position caused by the eye's continuous random walk.
By integrating these steps, the decoder maintains a dynamic estimate of both what the stimulus looks like and where it is currently located on the retina.
Biological Implementation in the Visual Cortex
A key contribution of this research is mapping this mathematical framework onto a biologically plausible neural network. The authors propose that the primary visual cortex (V1) is ideally structured to implement such a decoder. In their model, the "reaction" step corresponds to local, activity-dependent gain modulation within cortical columns. The "diffusion" step is implemented through lateral excitatory connections between neurons. These lateral connections, which are a hallmark of V1 architecture, effectively act as a prior that matches the statistics of fixational eye movements. This suggests that the brain does not need a separate motor signal to compensate for jitter; it can infer the movement trajectory directly from the spatial and temporal correlations of incoming retinal spikes.
Results and Human Performance
The performance of the Markov Decoder was evaluated using simulated retinal spikes generated from a model of the foveal cone mosaic and compared against human psychophysical data from a fine-orientation discrimination task. In this task, participants distinguished between tiny horizontal and vertical bars.
The Markov Decoder achieved approximately 100% accuracy on 1x2 arcminute bars and 70% on 0.5x1 arcminute bars, closely mirroring human performance levels. In contrast, "naive" decoding strategies—such as those assuming the eye is perfectly still or those that ignore the spatial correlation of movements—performed significantly worse. The model also demonstrated that the brain requires approximately 200–500 ms of temporal integration to accumulate enough evidence to "lock onto" the stimulus trajectory and achieve peak accuracy.
Discussion
This study provides a compelling theoretical account of how the human visual system maintains high acuity despite incessant eye motion. By framing vision as a process of dynamic inference rather than simple template matching, the Markov Decoder explains how the brain can leverage the inherent structure of retinal signals to overcome noise and uncertainty. The proposed neural implementation offers a functional explanation for the extensive lateral connectivity found in V1, suggesting these connections are fundamental to our ability to see the world in high resolution. While the current model focuses on simple oriented stimuli, it provides a foundation for understanding how the brain processes more complex visual scenes under the constant motion of natural viewing.
Summary updated