Hearing Sensor Fatigue: Perceptual Stress Tests for Speech Synthesized from Wearable Sensors

Skin-mounted sensors can now track throat vibration and facial muscle activity well enough to drive speech, which matters most for people who have lost their voice. These sensors wear out during use, though. Stretchable sensors drift, show hysteresis (their reading depends on whether strain is rising or falling), crack, and lose skin contact after thousands of deformation cycles. Meanwhile the models that turn articulator motion into audio are trained and scored on clean laboratory recordings, using word error rate and spectral distance. Those scores are known to be poor proxies for what listeners actually hear. Nobody has asked what a fatiguing sensor sounds like. Does drift make speech sound unnatural, do cracks make it sound noisy, and does lost contact make it sound choppy? Nobody has asked either whether the methods meant to tolerate degradation are ranked correctly when we only check word error rate.

This note proposes a benchmark and a listening protocol that answer both questions at a cost a single lab can afford. We inject physically grounded degradations into existing articulatory and muscle-signal speech corpora. Listeners score the resulting audio along three perceptual dimensions that have been validated for synthetic speech. Each listening trial is allocated adaptively, so that ratings go where they most change the conclusions. Uniform listening panels are expensive, and that expense is why this field keeps skipping subjective evaluation.

Core Claim

We make two claims, and each can be falsified. The first is that different sensor failure modes damage different perceptual dimensions. The three dimensions are those of Perceptual quality dimensions of text-to-speech systems, which factor-analyzed 16 rating scales and found that synthetic speech quality splits into Naturalness, Disturbances and Temporal Distortions. These correlate with overall quality at 0.806, 0.469 and 0.560 respectively. We expect slow baseline drift and hysteresis to mainly erode Naturalness, because they corrupt prosody and the shape of articulator trajectories. We expect crack-induced resistance spikes to show up as Disturbances such as hiss or raspiness, and contact loss to show up as Temporal Distortions such as interruptions. As a consequence, rankings of robustness methods by word error rate or mel-cepstral distortion will disagree with rankings by listeners, especially on Naturalness.

The second claim is that treating the listening test as a budget-allocation problem gives the same conclusions with substantially fewer ratings. This draws on the ranking-and-selection literature, which asks how to split a fixed sampling budget so that the best options are identified correctly.

The contribution is therefore evaluation methodology, not a new decoder. Dynamic allocation is used in two places. Its main role is to allocate listener effort. Its secondary role is as one of the robustness methods under test. Fatigue defines the stress conditions, and modular design is the infrastructure that makes the factorial study cheap to run.

What Exists and What Is Missing

Wearable speech sensors are already fatigue-tested, but only against classification targets. A Practical Strain Sensor Based on Ecoflex/Overlapping Graphene/Ecoflex Sandwich Structures reports a gauge factor of 715 and stability over 1,000 cycles, but separates only three spoken words. Ti3C2Tx Composite Aerogels Enable Pressure Sensors for Dialect Speech Recognition survives 6,000 cycles with 13.69% hysteresis and a recovery time under 72 ms, and classifies dialects at 96.2% accuracy. The epidermal sEMG tattoo-like patch plays back canned clips when it recognizes a command. These papers motivate our study, because they show degradation is real and measurable. They also constrain it, because their reported hysteresis, recovery time and cycle counts set the parameter ranges of our degradation models. None of them synthesizes continuous speech or measures perceived quality.

Continuous synthesis from articulator or muscle signals exists, but it assumes clean input. Deep Speech Synthesis from Articulatory Representations (HiFi-CAR) maps electromagnetic articulography straight to waveforms. Articulography is a lab system that tracks tongue, jaw and lip coils. HiFi-CAR reaches 18.5% word error rate on the mngu0 corpus and a mel-cepstral distortion of 3.26 on synthetic articulatory data. Digital Voicing of Silent Speech converts 8-channel surface muscle recordings to audio, at 3.6% word error rate on a closed vocabulary and 68.0% on an open one. We reuse both as the synthesizers under test. Neither reports perceptual dimensions, and neither models sensor wear.

The perceptual gap is documented directly. Production of phrases by mechanical models of the human vocal tract states that naturalness was "not quantitatively evaluated." Other speech papers likewise defer or skip listening tests (Impact of durational outlier removal, CreakVC). DMOSpeech optimizes intelligibility and speaker losses directly and reports "mode shrinkage," where prosodic variety drops as word error rate falls. That observation is a warning that objective gains can hide naturalness losses, and it is exactly the kind of disagreement our second claim predicts.

For the allocation side we borrow methods from two papers. Optimal Computing Budget Allocation for Data-Driven Ranking and Selection supplies the allocation rule for listening trials. CBCC3 allocates optimization budget to whichever subproblem recently improved most, using a small exploration probability ($p_t=0.05$). It supplies the template for an online channel-gating decoder.

CBCC3 found that rare exploration suffices because each component's usefulness changes smoothly. Sensor failure is abrupt, so CBCC3's finding may not transfer, and we test that rather than assume it. For infrastructure, OmniEvaluator splits evaluation into stages joined by an intermediate data format, so that $N$ backends and $M$ benchmarks connect with $O(N+M)$ glue code rather than $O(N\times M)$. Its judges are explicitly text-only. We extend the same design to sensor streams and to human perceptual judges.

Research Program

Degradation library. We build a library of parameterized degradation operators applied to sensor channels as a function of accumulated cycle count. There are four operators:

  • slow baseline drift;
  • rate-dependent hysteresis, with its loop width calibrated to the reported 13.69%;
  • sparse resistance spikes with recovery times up to about 72 ms, standing in for micro-cracking;
  • intermittent channel dropout, standing in for delamination or contact loss.

Each operator has a severity schedule indexed by cycles, covering roughly 1,000 to 6,000 cycles so that it stays within the ranges actually reported. We will be explicit about one limitation. The mngu0 data come from articulography coils, not stretchable sensors, so degradations applied there are an analogue for future wearable kinematic sensors rather than a replica. The muscle-signal corpus is physically closer to skin-mounted patches.

Modular harness. We build a harness with five swappable stages: sensor stream, degradation injector, decoder, vocoder and judge. They share one intermediate record holding the audio, the condition labels and the provenance. This lets every system meet every degradation and every judge without per-pair scripts.

Listening protocol. Listeners rate each stimulus on six scales chosen to load on the three factors, two per factor, plus an overall quality score. Because the original factor structure was found on German text-to-speech, we first re-run the factor analysis on our English stimuli. If a new factor appears, for instance a voicing or buzz dimension specific to sensor-driven synthesis, that counts as a finding rather than a failure.

Adaptive allocation. Each cell (robustness method × degradation type × severity) is treated as an option whose mean factor score must be ranked. After a uniform warm-up, the next block of trials goes to the cells whose extra ratings most increase the estimated probability that the rankings are correct. This is the same reasoning used for allocating simulation replications.

Baselines

Robustness methods and evaluation protocols have separate baselines, and both need to be fair.

For robustness, the do-nothing reference is the published synthesizer trained only on clean data. That means HiFi-CAR on mngu0 and the Gaddy–Klein transfer model on their muscle-signal data. On clean input they should reproduce the reported 18.5% word error rate on mngu0 and 3.6% / 68.0% closed / open on the muscle data. We consider any reproduction more than a few points off a failed replication.

The strongest competitor is the same architecture trained with our degradation operators as random data augmentation, including random channel dropout. It is the strongest because it needs no runtime machinery and directly targets the test conditions. Every method gets the same training data, the same augmentation operators and the same number of tuning trials.

The third baseline is a running per-channel baseline subtraction with spike clipping, which is cheap signal-processing hygiene.

The fourth is a contribution-gated decoder. It uses CBCC3-style recent-gain feedback to down-weight or skip channels whose contribution to decoder confidence collapses. Its ablation fixes the exploration probability at zero, which isolates whether exploration is needed to catch abrupt failures.

The gated decoder "beats" augmentation only if it is better on at least one perceptual factor at high severity, with no loss on the others.

For evaluation, the naive reference is a uniform listening design that spreads trials equally across cells. The strongest existing method is OCBA-style allocation in its standard single-objective form, which ranks by overall quality only. Our variant allocates jointly across the three factors. On Lacuna I found no reported result for OCBA applied to listening tests or speech synthesis, so there is no published number to beat. The comparison is internal and uses a dense gold-standard rating set, described below. Objective metrics serve as a further evaluation baseline: word error rate from an off-the-shelf recognizer, and mel-cepstral distortion.

Adaptive allocation wins if it matches uniform allocation's probability of correct ranking at half the trials or fewer.

Why not just…?

Why not just report word error rate and mel-cepstral distortion? These metrics are cheap and reproducible, and if they rank robustness methods the same way listeners do, the listening apparatus is unnecessary. The prior evidence points the other way. Naturalness is the factor most tied to overall quality (0.806), and DMOSpeech shows that improving intelligibility can shrink prosodic variety. Still, this is an empirical question. We compute Kendall's rank correlation between objective rankings and each factor's ranking across severities. If it stays high, say above 0.8 everywhere, we will report that objective metrics suffice for this regime.

Why not just train with degradation augmentation? This is the most serious threat to the gated decoder. Augmentation could plausibly match it, since fixed-capacity models often learn to ignore corrupt channels on their own. We include augmentation as the main robustness baseline for that reason. The core claims of the note survive either outcome, because they concern how failures map to perception and how to evaluate cheaply. If augmentation dominates, we drop gating as a recommendation.

Why not just run a bigger uniform listening test, or crowdsource it? A larger budget always works. But the factorial space here is large: two corpora, four or five methods, four degradation types and several severities. At that size, uniform coverage is exactly what makes labs defer listening tests. The retrospective-resampling experiment below measures how many ratings the adaptive design saves. If the savings fall below about 25%, uniform allocation is the honest recommendation.

Why not just use an automatic quality predictor as the judge? The research file contains no automatic quality-prediction model and no reported accuracy for one on sensor-driven speech. Any such predictor trained on text-to-speech output has never seen crack spikes or drift artifacts. We will plug one into the harness's judge slot as a secondary analysis, and report how well it agrees with each human factor score per degradation type. Disagreement concentrated in one factor would identify where automatic judges cannot yet stand in for listeners.

Why not just build tougher sensors? Better materials are complementary, not an alternative. Even the most durable sensor in the file still shows 13.69% hysteresis. Because severity in the benchmark is indexed by cycle count, materials groups can read off how much durability gain each perceptual factor needs, which hardware-only evaluation cannot provide.

Proposed Experiments

Dense gold-standard rating set. In the pilot phase, every cell receives dense ratings from a fixed listener pool on mngu0 and on the closed-vocabulary muscle-signal task, with natural speech and clean synthesis as anchors. This set serves as ground truth. Allocation policies are then compared by replaying them against it: each policy draws ratings from the stored pool, and we record its probability of correct ranking as a function of trial budget. This compares policies fairly at no extra listening cost.

Factor selectivity. We fit a mixed-effects model with listener and utterance as random effects. The model tests for an interaction between degradation type and perceptual factor. The prediction is that drift and hysteresis load mainly on Naturalness, spikes on Disturbances, and dropout on Temporal Distortions. The selectivity claim is falsified if every degradation lowers all three factors in proportion, that is, if the interaction is null.

Ranking disagreement. We compute Kendall's rank correlation between rankings by objective metrics and by each factor, as a function of severity. We expect the disagreement to grow with cycle count and to be largest on Naturalness.

Exploration under abrupt failure. For the gated decoder, we compare exploration probabilities of 0, 0.05 and higher values under sudden-dropout schedules. This tests whether CBCC3's finding that little exploration is needed carries over from smooth optimization landscapes to abrupt hardware failure.

Cost. We measure the trial budget at which adaptive allocation reaches 95% correct-ranking probability, against uniform allocation. A saving of at least half is the headline target, and below a quarter we would recommend uniform designs.

Together these experiments say whether wearable speech synthesis can be evaluated by ear at practical cost, and whether doing so changes which robustness methods the field should adopt.