TL;DR
This paper investigates whether pupil dilation can effectively measure cognitive load when listening to text-to-speech (TTS) systems, both in quiet and noisy environments. The study refines previous methods and introduces new experiments, concluding that pupil dilation primarily reflects attention in quiet conditions but shifts to measuring listening effort in noisy environments as signal quality degrades.
What Problem Does This Paper Address?
As text-to-speech (TTS) systems become more prevalent in everyday applications, understanding their impact on human cognitive processing is crucial. High cognitive load, especially in challenging listening conditions like noisy environments, can lead to fatigue and other negative consequences. While previous research has used pupil dilation as an indicator of cognitive effort, there have been uncertainties about what exactly it measures. This paper aims to clarify whether pupil dilation reflects true cognitive load or other factors like attention, particularly when listening to synthetic speech under varying noise levels. The authors also address limitations in previous statistical analyses by employing growth curve analysis (GCA) for time-series pupil dilation data.
Methods
The study builds upon previous work by re-analyzing existing data and conducting new experiments. A key methodological advancement is the use of Growth Curve Analysis (GCA). Unlike traditional methods like ANOVA, GCA is better suited for analyzing time-series data, allowing for a more nuanced understanding of how pupil dilation changes over time.
The experiments involved:
- Experiment 1 (Quiet Condition): Re-analysis of data from a previous study using GCA to examine pupil dilation when listening to TTS in quiet. This experiment focused on refining the analysis of pupil dilation patterns.
- Experiments 2 & 3 (Noisy Conditions): New experiments where participants listened to TTS in the presence of speech-shaped noise at two different signal-to-noise ratios (-1dB and -3dB). These experiments were designed to investigate how pupil dilation behaves under increasing listening difficulty.
Participants were native English speakers with no reported hearing problems. Speech stimuli included natural speech and speech generated by four different TTS synthesizers (Hybrid, Unit Selection, HMM, and LQ-HMM). Pupil size was monitored using an eye tracker while participants listened to the stimuli. After each block of listening, participants provided self-reported measures of cognitive load, motivation, and naturalness.
Datasets
The study utilized existing datasets of natural and synthetic speech. The synthetic speech was generated by four synthesizers from the 2011 Blizzard Challenge. The natural speech was from the same speaker as the synthetic speech. No new datasets were introduced.
Key Insights and Ideas
- Quiet vs. Noise is Key: The most significant finding is the distinction between what pupil dilation measures in quiet versus noisy conditions. In quiet, it seems to reflect attention and engagement rather than pure cognitive load.
- Noise Increases Effort: In noisy environments, pupil dilation reliably increases as the signal-to-noise ratio (SNR) decreases, indicating a rise in listening effort. This suggests that pupil dilation can be a valid measure of cognitive load when the listening task is sufficiently demanding.
- Quality Matters, But Not Always as Expected: While higher quality TTS generally leads to lower pupil dilation in quiet (suggesting less effort/more engagement), the relationship in noise is more complex. Poor quality systems might reach a "ceiling" in perceived cognitive load even at easier SNRs.
- GCA Unlocks Nuance: The application of Growth Curve Analysis provided a more detailed understanding of the temporal dynamics of pupil dilation, revealing differences in peak shape and slope between TTS systems that might have been missed with simpler statistical methods.
Key Results
- Quiet Condition: Pupil dilation in quiet conditions did not directly correlate with listening effort but rather with attention and engagement. Higher quality TTS led to flatter pupil dilation curves, suggesting greater engagement.
- Noisy Conditions (-1dB and -3dB SNR): Increased pupil dilation was observed as SNR decreased, indicating a rise in listening effort. This effect was more pronounced for higher quality TTS systems.
- System Differences: GCA revealed statistically significant differences in pupil dilation patterns (intercept, linear, quadratic, and cubic terms) between various TTS systems, particularly in noisy conditions. For instance, in the -3dB SNR condition, the Unit Selection system showed the sharpest peak, while the Natural condition had the flattest.
Limitations and Future Work
- Quiet Condition Interpretation: The authors acknowledge that the interpretation of pupil dilation in quiet conditions remains somewhat uncertain, with attention and engagement being the primary candidates rather than cognitive load.
- Self-Reported vs. Physiological Measures: There were instances where self-reported cognitive load did not perfectly align with pupil dilation measures, especially in noisy conditions with poor quality TTS. This discrepancy warrants further investigation.
- Future Directions: The authors suggest that further research could explore the interplay between different types of noise, more diverse TTS systems, and a wider range of participant demographics. Investigating the specific acoustic features of TTS that contribute to cognitive load and attention would also be valuable.
Summary updated