TL;DR
This paper introduces an adaptive sampling method for speech perception experiments that significantly reduces the number of trials needed without sacrificing accuracy. By intelligently selecting which stimuli to resample based on a logistic regression model of listener responses, experiments can be shortened by about a third, making them more efficient for both researchers and participants.
What Problem Does This Paper Address?
Speech perception research often involves creating synthetic stimuli that vary along multiple acoustic dimensions (like vowel formants or duration) to understand how listeners categorize sounds. However, the number of unique stimuli grows exponentially with each added dimension. To accurately model listeners' perception, especially around category boundaries, a large number of responses per stimulus is typically required. Collecting this data is time-consuming, leading to participant fatigue and potential dropout, particularly in longitudinal studies. This paper tackles the challenge of efficiently collecting high-resolution perceptual data by reducing the total number of trials needed in an experiment.
Methods
The core of the proposed method is an adaptive sampling procedure. After an initial pass where all stimuli are presented a few times, a logistic regression model is fitted to the collected response data. This model predicts the probability of a listener choosing a particular category for each stimulus.
The key innovation lies in how stimuli are selected for resampling in subsequent rounds:
- Initial Pass: All stimuli are presented a fixed, small number of times (e.g., twice).
- Model Fitting: A logistic regression model is fitted to this initial data.
- Error Calculation: The difference (error) between the model's predicted response probabilities and the observed response proportions is calculated for each stimulus.
- Stimulus Selection for Resampling: Instead of resampling randomly or uniformly, the procedure prioritizes stimuli with the largest errors. This focuses data collection on areas where the model is less certain, typically near category boundaries. To avoid completely neglecting "easy" stimuli, a stochastic selection process is used: stimuli are ranked by error, and a random score is added to their error score. Stimuli exceeding a median threshold are selected for resampling.
- Iteration: Steps 2-4 are repeated multiple times.
This adaptive approach ensures that stimuli far from category boundaries, where responses are consistently one category, are sampled less often, while stimuli near boundaries, where responses are mixed, are sampled more frequently to refine the model's accuracy in those critical regions.
Datasets
The adaptive sampling procedure was initially developed and described using a synthetic vowel stimulus set designed to investigate the perception of English /i/, /1/, /e/, /ε/, and Spanish /i/, /ei/, /e/. This set comprised 90 stimuli varying along three acoustic dimensions: duration, F1-F2 space, and vowel inherent spectral change (VISC).
For evaluation, the authors generated 100 simulated response sets based on a logistic regression model fitted to original test data. These simulated datasets allowed them to compare the performance of their adaptive sampling method against a standard, full-sampling approach.
Key Insights and Ideas
- Focusing on Uncertainty: The most crucial insight is that not all stimuli require the same amount of sampling. By focusing resampling efforts on stimuli where listener responses are most variable (i.e., near category boundaries), significant efficiency gains can be achieved.
- Adaptive Sampling as a Smart Shortcut: The procedure acts like a smart researcher who knows where to look for the most informative data, rather than blindly collecting everything.
- Error-Driven Resampling: Using the error between predicted and observed response probabilities as the primary criterion for resampling is effective because high error often indicates a stimulus is near a perceptual boundary.
- Stochastic Selection for Robustness: Including a random component in stimulus selection for resampling ensures that even stimuli with lower error scores have a chance to be heard. This provides listeners with "anchors" and can help maintain participant engagement.
- Mistake Amelioration: The error-based selection criterion has a beneficial side effect: accidental button presses (mistakes) by participants can inflate a stimulus's error score, making it more likely to be resampled and thus potentially correcting for the erroneous response.
Key Results
Simulations indicated that the adaptive sampling procedure could reduce the number of trials by approximately one-third without substantially affecting the accuracy of the results. Specifically:
- The Sum of Absolute Errors in Proportions (SAEP) between the adaptive and full-sampling models was not significantly different in simulations.
- While some logistic regression coefficient values showed statistically significant differences, the magnitude of these differences was small (mean differences generally less than 4.5%).
- In practice, participants in the described study completed the perception experiment in 20 to 40 minutes, a significant reduction from the estimated half-hour needed for just four responses per stimulus.
- Participant retention was very high, with only 3 out of 95 participants dropping out after the first session in a multi-session study.
Limitations and Future Work
- Model Complexity: The authors note that while a simple linear logistic regression model was used for stimulus selection during the adaptive procedure, the best-fitting model for the actual data might be more complex (e.g., involving non-linear relationships or discrete levels for certain acoustic features). However, they argue that using an underfitted model for selection is preferable to overfitting, as it leads to more resampling rather than missing important patterns.
- Further Efficiency Gains: The paper suggests that additional reductions in sampling could be achieved by:
- Using stimulus sets with higher acoustic density or a greater number of samples per stimulus.
- Employing a multi-stage approach, starting with a low-density matrix and then switching to a high-density matrix using a different selection criterion (e.g., furthest from 0 or 1).
- Generalizability: While simulations were performed, further testing on a wider range of simulated data and real-world datasets could confirm the robustness of the method across different perceptual tasks and acoustic spaces.
Summary updated