Introduction
In fields ranging from neuroscience to cognitive science, researchers often rely on "simulator-based" models to explain complex phenomena. While these models can easily generate synthetic data, they frequently lack an analytical likelihood function. This makes it difficult to determine the probability of observing specific real-world data points, a process essential for parameter estimation and model comparison.
Standard approaches to this problem often introduce systematic biases that can lead to incorrect scientific conclusions. This paper introduces Inverse Binomial Sampling (IBS), a simple yet powerful method to estimate the log-likelihood of simulator-based models without the systematic biases found in traditional simulation techniques.
The Problem: The Bias of Fixed Sampling
In modern statistical inference, the likelihood function is the primary tool for evaluating how well a model’s parameters explain observed data. When the likelihood is intractable, researchers often use "likelihood-free" methods or simple simulation. In a typical fixed-sampling approach, one draws a fixed number of samples ($M$) from the model and counts how many match the observed data to estimate the probability $p$.
However, most optimization and model selection routines require the log-likelihood. Because the logarithm is a non-linear transformation, the log of an average is not equal to the average of the logs (a consequence of Jensen's inequality). Consequently, fixed-sampling estimates of the log-likelihood are systematically biased. If a model predicts an event is very rare, a fixed-sampling approach might never observe it, leading to a $\log(0)$ result—an infinite bias that can derail optimization algorithms and model comparisons.
The IBS Method: Sampling Until Success
The authors propose a fundamental shift from "fixed sampling" to "inverse sampling." Instead of drawing a pre-determined number of samples, the IBS policy dictates that one should keep drawing samples from the simulator until exactly one match (a "hit") with the observed data point is found.
The Estimator
The log-likelihood estimate for a data point is calculated based on $K$, the number of samples required to achieve that single hit. The estimator is derived from a sum of reciprocal integers, related to the harmonic series or the digamma function.
Key Properties
- Unbiasedness: Unlike fixed sampling, the expected value of the IBS estimator is exactly the true log-likelihood. This "uniform unbiasedness" is a critical theoretical advantage for model fitting.
- Bounded Variance: The variance of the IBS estimate is uniformly bounded, never exceeding $\pi^2/6 \approx 1.645$, regardless of how low the true probability of the event is. This ensures stability even when dealing with rare events.
- Multi-fidelity: To further reduce noise, the process can be repeated $R$ times and the results averaged, allowing researchers to trade computational time for increased precision.
Empirical Evaluation
The authors evaluated IBS across three model-fitting problems of increasing complexity within computational cognitive science:
- Orientation Discrimination: A psychometric task where participants judge the tilt of a line. This task has an analytical solution, providing a ground-truth baseline to verify the accuracy of IBS.
- Change Localization: A task requiring participants to identify which of six items changed orientation. This requires numerical integration for an exact solution, representing a more challenging test case.
- Four-in-a-Row Game: A complex sequential decision-making task (a variant of tic-tac-toe). The likelihood here is truly intractable, representing the primary use case for IBS in real-world research.
Results and Discussion
Across all tasks, IBS demonstrated significant advantages over traditional methods:
- Superior Parameter Recovery: IBS produced lower Root Mean Squared Error (RMSE) in estimated parameters compared to fixed sampling, even when the fixed-sampling method used the same average number of samples.
- Accurate Model Evidence: IBS recovered maximum log-likelihood values within 1–2 points of the ground truth. In contrast, fixed sampling remained strongly biased even with a large number of samples, which could lead to selecting the wrong model during scientific inquiry.
- Computational Efficiency: IBS automatically allocates more effort to "hard" trials (low-probability events) and less to "easy" ones. This adaptive nature makes it more efficient than fixed-sampling methods that apply the same computational budget to every trial.
- Compatibility with Optimization: Because the total log-likelihood is a sum of independent estimates, the Central Limit Theorem ensures the total estimate is approximately Gaussian. This makes IBS an ideal partner for optimization algorithms like Bayesian Adaptive Direct Search (BADS), which are designed to handle Gaussian noise.
Limitations and Future Directions
While IBS is a robust "out-of-the-box" solution, it does have limitations. In its current form, it requires discrete model outputs. While the authors propose an "Approximate IBS" (AIBS) for continuous data using distance metrics and tolerance thresholds, this area requires further theoretical development.
Additionally, if a model parameter is extremely poor—making the probability of a match near zero—IBS could theoretically sample indefinitely. The authors recommend mitigating this "hang risk" by including a small "lapse rate" (random error rate) in the model or implementing an early-stopping threshold. Future work involves integrating IBS with advanced Bayesian inference techniques such as Markov Chain Monte Carlo (MCMC) and Variational Inference.
Summary updated