Phoneticians keep finding acoustic differences that nobody has checked whether people can hear. Examples include stop consonants whose timing cues overlap, vowels that differ by a few tens of milliseconds, and synthetic speech whose spectral slope deviates from natural speech by several dB. Increasingly, the stand-in for a listening test is a machine-learning classifier trained on the recordings. The authors who use such classifiers warn that a classifier's accuracy is not human perception. We propose a different stand-in that can be checked against real listeners. We would build a small mechanistic simulated listener: an auditory front end with internal noise, followed by a decision process that accumulates acoustic cues as they arrive in time and produces a choice and a reaction time. We would then fit its parameters to a short, cheap listening experiment.
The listener simulator has no tractable likelihood. Internal noise is injected into a time-varying auditory representation, and the decisions race against each other with cue-dependent timing. For this reason we fit it with likelihood-free inference, also called simulation-based inference. These methods infer parameters by running the simulator many times rather than evaluating a probability formula. The fitted posterior over each listener's cue weights and noise levels can then be applied to production data. For any contrast measured in a corpus, it gives a predicted, uncertainty-quantified answer to the question "would this group of listeners hear this?" The decisive test is whether those predictions match held-out human discrimination of natural corpus tokens better than classifier proxies do.
Core Claim
A perception experiment run once, on a controlled set of resynthesized stop-voicing continua, can calibrate a listener model. That calibrated model should predict perception of natural and synthetic tokens it never saw, and it should predict it better than discriminative classifiers trained on production data. Likelihood-free inference earns its place here for a specific reason. Making the listener mechanistic is what should make it transfer: noise sits on the auditory signal rather than on hand-measured cues, and cues are integrated in the order they occur in time. That mechanism is exactly what makes the likelihood intractable.
We restrict the domain to stop voicing. The file contains three otherwise disconnected cases of unvalidated acoustic findings in that domain, and one listener model can address all three. The first is that secondary voicing cues carry high classifier importance in Malaysian English (Relative cue weighting in multilingual stop voicing production). The second is that WaveNet-based TTS lowers $F_0$ onset after voiceless stops and flattens consonant spectral tilt (Production characteristics of obstruents in WaveNet and older TTS systems). The third is that word-medial Tohoku Japanese stops show fully overlapping voice onset time (VOT, the delay between the stop's release and the start of vocal-fold vibration) (Neutralization of Voicing Distinction of Stops in Tohoku Dialects of Japanese).
Position Relative to Prior Work
Three papers motivate the proposal by documenting the gap. The French initial-strengthening study (Are IP Initial Vowels Acoustically More Distinct?) uses linear discriminant analysis on formants and a convolutional network on spectrograms as perception proxies, and its authors explicitly disclaim them as measures of hearing. The Malaysian cue-weighting study derives random-forest Gini importances from production, again without listeners. The TTS study shows that high aggregate opinion scores coexist with systematic segmental deviations, but it never tests their audibility.
All three infer perceptual relevance from how separable the production data are. We infer it from a model fitted to listeners.
The methods come from the simulation-based inference literature. Benchmarking Simulation-Based Inference compares rejection ABC, sequential Monte Carlo ABC and neural estimators of the likelihood, posterior and density ratio. It reports that sequential neural posterior and likelihood estimation reach classifier two-sample test (C2ST) scores near 0.5 on Two Moons at $10^5$ simulations, while rejection ABC stays clearly worse. That result constrains our design: we will not rely on classical rejection ABC with hand-chosen summaries.
The reviews Summary Statistics in Approximate Bayesian Computation and Semi-automatic ABC explain why low-dimensional, learned summaries are needed. Diagnostic tools for approximate Bayesian computation using the coverage property supplies the coverage checks we use to validate posteriors when no ground truth exists.
The closest methodological precedent is ABrox. It applies ABC model comparison to multinomial processing tree models of categorical responses in a weapon-misidentification task. It works with abstract discrete cognitive models and never touches acoustic stimuli or auditory front ends. We borrow its idea of likelihood-free model choice between cognitive architectures but apply it to a continuous, signal-driven listener.
We deliberately do not pursue articulatory inversion of vocal-tract simulators (as in Acoustic-to-Articulatory Inversion with Deep Autoregressive Articulatory-WaveNet). That direction inverts production. Inverting production does not answer whether a difference is heard.
Research Program
The listener simulator. It takes a stimulus waveform. It passes the waveform through a fixed auditory filterbank with compressive nonlinearity, then adds Gaussian internal noise with a per-listener level. From this noisy representation, simple fixed detectors read out five cues as they become available within the syllable: closure duration, burst intensity, VOT, onset $F_0$ and onset $F_1$. Each cue feeds evidence into two competing accumulators, one for "voiced" and one for "voiceless," with a per-listener weight. Evidence from a cue enters only after that cue occurs in time.
A trial ends when one accumulator reaches threshold or when the listener lapses. The per-listener parameters $\theta$ are the cue weights, the internal noise level, a response bias, the threshold, non-decision time and lapse rate, about ten in all. Each simulator run returns a choice and a reaction time for one stimulus.
Calibration experiment. We will resynthesize continua from natural tokens in the Malaysian multilingual corpus. In these continua VOT, onset $F_0$, closure duration and burst intensity are varied factorially, partly in conflicting combinations, because cue conflicts are what make weights identifiable. Malaysian English listeners give two-alternative identification responses with reaction times, roughly 400 trials per listener. A parallel group of native English listeners does the same task on matched continua.
Inference. We train neural posterior estimation on simulated listeners, amortized over the stimulus set. A learned embedding network compresses each listener's trial-level choices and reaction times, conditioned on stimulus identity. Amortization makes per-listener posteriors cheap to obtain and lets us run simulation-based calibration and coverage diagnostics on thousands of synthetic listeners with known parameters before touching human data. Using ABC-style model choice in the spirit of ABrox, we compare the full model against variants without temporal cue ordering and without front-end noise.
From listener posterior to perceptual validation. For a production contrast between token distributions $A$ and $B$, such as natural versus TTS /t/, we define a predicted perceptual distinctiveness:
$$D(A,B) = \mathbb{E}_{\theta \sim p(\theta \mid \text{data})}\left[\, \left| P_\theta(\text{voiced} \mid a) - P_\theta(\text{voiced} \mid b) \right| \,\right], \quad a \sim A,\ b \sim B,$$
Here $P_\theta(\text{voiced} \mid a)$ is the simulated listener's probability of responding "voiced" to waveform $a$. The expectation runs over the posterior over $\theta$ and over sampled token pairs. The spread of the integrand gives a credible interval. A difference "lacks perceptual relevance" when that interval sits near zero. This rules out treating a significant acoustic difference that the model says listeners would not hear as a perceptual finding.
Baselines
The naive reference point is the inference phoneticians currently make by default: a contrast counts as perceptible if the production difference is statistically significant. We will operationalize it as a mixed-model test on the acoustic cues, with effect size as the predicted perceptual difference. Beating it means a higher rank correlation between predicted and observed listener discrimination across our held-out contrasts.
The strongest existing proxy is classifier separability. This means linear discriminant analysis and a spectrogram convolutional network as in the French vowel study, and a random forest on the nine production cues as in the Malaysian study. For reference, the French study reports overall classification accuracy of 68% (LDA) and 69% (CNN) for intonational-phrase-initial vowels, against 63% and 60% for word-initial vowels. For random-forest Gini importances in the Malaysian corpus, no numeric values are reported in the file, and no reported result was found on Lacuna for any classifier's agreement with listener data. These classifiers will be trained on exactly the corpus tokens our model never sees. Their predicted distinctiveness is held-out accuracy separating $A$ from $B$, and their cue weights are their importances. They get the same hyperparameter search budget as our embedding network. Beating them means better prediction of held-out human AX discrimination and identification of natural tokens, measured by per-token response-proportion log-likelihood and by correlation of predicted with observed $d'$.
The strongest perception-side competitor is a tractable psychometric cue-weighting model: a Bayesian logistic regression of identification responses on measured cue values, ignoring reaction times. It is fitted to the same calibration data with the same priors. Beating it requires better transfer to natural and TTS tokens. Matching it on the calibration data alone does not count.
Three ablations isolate the key ingredient: removing front-end noise (cues fed in directly), removing temporal ordering of cues, and removing reaction times from the inference.
For the inference engine itself, we benchmark neural posterior estimation against sequential Monte Carlo ABC with semi-automatic summaries at matched simulation budgets of $10^3$, $10^4$ and $10^5$. On synthetic listeners we score both with C2ST. The file gives no reported C2ST value for this task; it reports near-0.5 C2ST for sequential neural methods only on Two Moons.
Why not just…?
Why not just trust classifier accuracy and feature importance from production data? The papers that use these proxies say they are not perception: the French vowel authors and the Malaysian cue-weighting authors both caution against reading classifier separability as human hearing. A classifier can also exploit cue combinations listeners ignore. The plausible failure case is a large consonant spectral tilt deviation, about −7 to −8 dB/log(Hz) in the TTS study, that may be inaudible in context. Our held-out comparison measures directly whether the classifier or the fitted listener better predicts human responses. If the classifier wins, that is itself a useful result: the proxies would be partly vindicated.
Why not just fit a logistic psychometric model, which has a closed-form likelihood and needs no simulation? This is the real threat. For single-cue continua it is the right tool, and our calibration data may be fit equally well by it. The argument for the mechanistic model is transfer. Natural and synthetic tokens differ from resynthesized continua in properties that measured cue values do not capture, such as cue reliability under spectral distortion. Placing noise on the auditory representation should capture that difference, while placing noise on measured cues should not. The front-end-noise ablation and the logistic baseline test exactly this claim. If the logistic model transfers as well, we will adopt it and drop the simulator. In that case likelihood-free inference would not be needed for this problem.
Why not just run a listening test for every contrast? Direct tests are the ground truth, and we run them on held-out contrasts to score every method. But each corpus finding currently needs its own experiment with matched listeners. The proposal's value is amortization: one calibration experiment per listener group, then predictions with uncertainty for every new acoustic difference. Prediction error on held-out contrasts tells us whether that amortization is trustworthy.
Why not just use rejection ABC with intuitive summaries such as mean accuracy and median reaction time? The SBI benchmark reports rejection ABC well behind sequential neural methods at matched budgets. The summary-statistics review shows that hand-picked summaries lose information about parameters, and cue weights would be among the first casualties because they are identified only through cue-conflict trials. We nonetheless include sequential ABC with semi-automatic summaries so that the gap is measured on this task rather than assumed.
Decisive Checks
The program is first tested on synthetic listeners. Posterior coverage must be nominal under the coverage diagnostic, and cue weights must be recoverable from 400 trials. If they are not, the experiment design is revised before any human data are interpreted.
The main test is held out. Beyond the calibration continua, Malaysian English and native English listeners do AX discrimination and identification on unmodified natural tokens from the Malaysian corpus and on matched natural-versus-WaveNet obstruent tokens from the Blizzard 2013 material. Neither the listener model nor the logistic baseline sees these tokens. The proposal succeeds if three things hold: predicted $D$ ranks contrasts by observed discriminability better than classifier separability does; it beats the logistic cue model on held-out log-likelihood; and the front-end-noise ablation loses most of that advantage.
Two secondary outcomes carry their own scientific value. One is whether Malaysian English listeners' inferred weights on onset $F_0$ and closure duration match the production-side random-forest importances or diverge from them. The other is whether the WaveNet $F_0$-onset lowering yields a credible interval on $D$ excluding zero. A stretch extension would recalibrate with Tohoku listeners and test whether word-medial stops with overlapping VOT are predicted, and observed, to be heard as the same.
References
- Relative cue weighting in multilingual stop voicing production
- Production characteristics of obstruents in WaveNet and older TTS systems
- Neutralization of Voicing Distinction of Stops in Tohoku Dialects of Japanese: Field Work and Acoustic Measurements
- Are IP Initial Vowels Acoustically More Distinct? Results from LDA and CNN Classifications
- Benchmarking Simulation-Based Inference
- Summary Statistics in Approximate Bayesian Computation
- Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC
- Diagnostic tools for approximate Bayesian computation using the coverage property
- ABrox—A user-friendly Python module for approximate Bayesian computation with a focus on model comparison
- Acoustic-to-Articulatory Inversion with Deep Autoregressive Articulatory-WaveNet