A Framework for Testing Identifiability of Bayesian Models of Perception

Luigi Acerbi, Wei Ji Ma, Sethu Vijayakumar

Introduction

While Bayesian models are excellent at describing human perception, they often suffer from a "degeneracy" problem: different combinations of mental priors, likelihoods, and loss functions can produce identical behavioral outputs. In psychophysics, researchers fit these models to human behavior to "read out" how the brain represents uncertainty (the likelihood), expectations (the prior), and goals (the loss function).

However, there is a fundamental inverse problem: the mapping from behavior back to these internal components is not unique. For example, a specific bias in a subject's response could be explained by a skewed prior or an asymmetric loss function. If a model is "non-identifiable," any conclusions drawn about a subject's internal mental state are essentially guesswork. This paper introduces a formal framework to test identifiability—the ability to uniquely recover these internal components from experimental data—allowing researchers to optimize experimental designs before they even collect data.

Methods: Modeling the Observer

The authors propose a two-part solution to tackle the identifiability problem:

  1. A Flexible Observer Model: They develop a generalized Bayesian observer model that uses a "mixture of Gaussians" approach to represent priors, likelihoods, and loss functions. This allows the model to approximate almost any shape (skewed, heavy-tailed, etc.) while remaining computationally efficient enough for large-scale simulation.
  2. Identifiability Mapping: To test a design, they define a "reference observer" (the ground truth). They then use Kullback-Leibler (KL) divergence to measure how much the predicted behavior of a candidate model differs from the reference. By using Markov Chain Monte Carlo (MCMC) to sample the "approximate expected posterior density," they can visualize the "identifiability landscape."
Generative and internal models of the observer

If the landscape has a sharp peak at the true parameters, the model is identifiable. If it shows long ridges or multiple peaks, the model is degenerate.

Case Studies

As a theoretical framework paper, the authors apply their method to two well-known "case studies" in psychophysics using simulated data:

  • Time Interval Estimation: Based on the "Ready-Set-Go" paradigm.
  • Speed Perception: Based on the "slow-speed prior" model.

Key Insights and Results

The "Anti-Correlation" Trap in Interval Timing

In time perception tasks, the authors found a strong anti-correlation between inferred sensory noise and motor noise. If an experiment does not measure motor noise independently, the model can "trade off" one for the other, leading to incorrect estimates of the subject's internal prior.

In the baseline experimental setup (BSL), the standard deviation of the recovered prior was consistently lower than the true value due to this sensory/motor noise degeneracy. However, by adding a separate condition to measure motor noise (MTR), the degeneracy was almost entirely removed.

Posterior distributions and primacy probabilities for interval timing designs

To quantify this, the authors introduced a "primacy probability" metric to rank experimental designs. In the timing task, the MTR design had a 79% probability of having the lowest reconstruction error, compared to only 6% for the baseline design.

Counter-Intuitive Design Flaws

Surprisingly, the analysis showed that experimental designs intended to "help" the subject—such as providing high-precision feedback that encourages a narrow loss function—can actually make the model less identifiable. A "better" observer isn't always a more "identifiable" one.

Robustness of Speed Perception

The model for speed perception, which assumes humans expect things to move slowly, turned out to be remarkably robust. Even when internal representations were decoupled from objective reality, the core parameters of the prior (such as the exponent of the power-law prior, $k_{prior}$) were recovered with high accuracy across different assumptions. This provides strong theoretical validation for previous empirical studies claiming humans have a heavy-tailed prior for slow speeds.

Limitations and Future Work

  • Dimensionality: The current framework focuses primarily on one-dimensional stimulus variables (e.g., duration, speed, orientation). Extending this to high-dimensional or categorical stimuli remains a computational and theoretical challenge.
  • Parametric Assumptions: While the mixture-of-Gaussians approach is highly flexible, it still assumes certain functional forms. The "true" mental representations might fall outside this basis.
  • Task Complexity: The framework currently handles estimation tasks well, but the authors note that two-alternative forced-choice (2AFC) tasks—which are ubiquitous in psychophysics—have identifiability properties that are still largely unknown and require future adaptation of this method.

Summary updated

Abstract

Bayesian observer models are very effective in describing human performance in perceptual tasks, so much so that they are trusted to faithfully recover hidden mental representations of priors, likelihoods, or loss functions from the data. However, the intrinsic degeneracy of the Bayesian framework, as multiple combinations of elements can yield empirically indistinguishable results, prompts the question of model identifiability. We propose a novel framework for a systematic testing of the identifiability of a significant class of Bayesian observer models, with practical applications for improving experimental design. We examine the theoretical identifiability of the inferred internal representations in two case studies. First, we show which experimental designs work better to remove the underlying degeneracy in a time interval estimation task. Second, we find that the reconstructed representations in a speed perception task under a slow-speed prior are fairly robust.
BibTeX
@inproceedings{acerbi2014framework,
  author = {Luigi Acerbi and Wei Ji Ma and Sethu Vijayakumar},
  title = {{A Framework for Testing Identifiability of Bayesian Models of Perception}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
}
Figure