TL;DR
This paper tackles the crucial problem of whether we can reliably infer the internal components (prior beliefs, encoding mechanisms, and loss functions) of Bayesian models of cognition from behavioral data. The authors demonstrate, both theoretically and empirically, that these components are generally identifiable, contrary to some prior pessimistic views, especially when data is collected across multiple levels of sensory noise. This work provides a vital theoretical foundation for reverse-engineering cognitive processes and offers practical guidance for designing experiments.
What Problem Does This Paper Address?
Cognitive science and neuroscience often use Bayesian decision theory as a normative framework to model how humans make decisions, particularly in perception. This framework posits that observers combine prior beliefs about the world with sensory evidence (likelihood) to form a posterior belief, and then use a loss function to decide on an action. A key challenge is determining if we can actually recover these underlying components (prior, likelihood, loss) from observed behavior. Previous work has often struggled with "degeneracy," where different combinations of these components can explain the same behavior, leading researchers to often assume some components are known. This paper addresses the fundamental question of whether these Bayesian model components are identifiable from behavioral data, and under what conditions.
Methods
The authors develop a rigorous mathematical framework to analyze the identifiability of Bayesian observer models. They model the neural encoding of a stimulus as a general nonlinear transformation with additive sensory noise. Their approach involves:
- Theoretical Analysis: Deriving theorems that establish conditions for identifiability. This includes analyzing the role of sensory noise levels and the relationship between different model components.
- Method of Moments: Utilizing this statistical technique to match observed response distributions with theoretical ones, allowing for the estimation of model parameters.
- Numerical Simulations: Generating synthetic datasets from known ground-truth models to test their theoretical predictions and assess model recovery under various conditions.
- Fitting Procedure: Employing a general fitting procedure that jointly estimates encoding, prior, motor, and sensory noise, and guessing rate by maximizing data likelihood. This procedure is applied to both simulated and real-world datasets.
Datasets
The paper utilizes existing published datasets for its applications:
- Orientation Estimation Data (de Gardelle et al., 2010): This dataset involves human observers reproducing visual orientations under varying stimulus presentation times (20ms, 40ms, 80ms, 160ms, 1000ms), which are used as proxies for different sensory noise levels.
- Time Interval Data (Remington et al., 2018): This dataset involves subjects reproducing time intervals, used to test identifiability in a different perceptual domain.
The paper also generates extensive synthetic datasets for simulations, varying parameters like the number of trials and noise levels.
Key Insights and Ideas
- Identifiability is Generally Achievable: Contrary to some prior assumptions, the paper demonstrates that Bayesian model components (prior, encoding, loss) are generally identifiable from behavioral data, especially under specific experimental designs.
- The Power of Multiple Noise Levels: The most crucial insight is that collecting data across multiple levels of sensory noise is key to overcoming model degeneracy and achieving reliable identifiability, particularly for the loss function. This is because the bias in responses scales non-linearly with noise magnitude in a way that can distinguish between different loss functions.
- Encoding is Relatively Easy to Identify: The encoding function (specifically, the Fisher information) can often be identified even with data from a single noise level, provided mild assumptions on smoothness and small sensory noise.
- Prior and Loss Function are Often Confounded: When only a single noise level is available, the prior and loss function can be systematically confounded, making it difficult to disentangle them.
- Special Cases of High Degeneracy: The paper identifies specific combinations of encoding (e.g., Weber's law) and prior (e.g., lognormal) that exhibit unusually high degeneracy, even with multiple noise levels. However, these are shown to be rare (measure zero) in the space of all possible models.
- Adaptation as a Solution: For these rare degenerate cases, adapting the system to different stimulus statistics (e.g., changing the prior) can help resolve the ambiguity.
Key Results
- Theoretical Guarantees: The paper provides theoretical guarantees for the identifiability of encoding and prior under mild conditions, and for the loss function when data from at least two noise levels is available.
- Simulation Validation: Extensive simulations confirm that model components can be reliably recovered when using multiple noise levels, even with moderate sample sizes (e.g., ~1000 trials).
- Experimental Data Support: Applications to real-world datasets demonstrate that the theoretical insights hold in practice. For instance, the encoding for visual orientation was consistently recovered across different stimulus presentation times (noise levels).
- Loss Function Identification: Combining data from multiple noise levels significantly improved the identifiability of the loss function exponent, which was difficult to determine from a single noise level. For example, with 2000 trials across two noise levels, the difference between best and worst-fitting exponents could be over 20, compared to much smaller differences with a single noise level.
- Efficiency of Noise Manipulation: The results suggest that manipulating the number of noise levels can be more effective for identifiability than simply increasing the number of trials.
Limitations and Future Work
- Exceptional Set of Models: While most models are identifiable, the paper acknowledges a rare "exceptional set" of models (e.g., Weber's law encoding with a lognormal prior) that remain difficult to identify even with multiple noise levels. Future work could explore strategies like adaptation to resolve these specific cases.
- Parametric Assumptions: While the theory aims for generality, practical fitting procedures may still rely on some assumptions (e.g., smoothness of priors and encodings).
- Generalizing to Other Tasks: The current work primarily focuses on estimation tasks. Future research could extend these identifiability results to other paradigms like two-alternative-forced choice (2-AFC) tasks, though simulations suggest 2-AFC might be less efficient.
- Optimizing Experimental Design: The paper suggests that understanding how to optimally design experiments (e.g., choosing the number and spacing of noise levels) for a given number of trials is an important direction for future research.
- Stimulus Noise: While the paper focuses on sensory noise, future work could explore the role of stimulus noise in model identification.
Summary updated