Intermittent Speech Recovery

Yu-Chen Lin, Tsun-An Hsieh, Kuo-Hsuan Hung, Cheng Yu, Harinath Garudadri, Yu Tsao, Tei-Wei Kuo

TL;DR

This paper introduces a novel Intermittent Speech Recovery (ISR) system designed to reconstruct speech signals captured by battery-free, energy-harvesting IoT devices. By combining frequency-domain interpolation, a complex-valued deep learning enhancement model, and a final signal combination stage, the authors achieved a massive 591.7% increase in speech quality (PESQ) and a 52.6% improvement in word error rate (WER) for downstream speech recognition.

What Problem Does This Paper Address?

The "Internet of Things" (IoT) is moving toward battery-free operation, relying on environmental energy harvesting (solar, thermal, or vibration). However, these energy sources are inherently unstable. As shown in Figure 2, these devices operate in cycles: they harvest energy until a threshold ($V_{on}$) is reached, perform tasks like recording speech, and then abruptly shut down when the voltage drops to $V_{off}$.

![Fig.

  1. (a) self-powered intermittent system (b) general voltage trace of self-powered intermittent systems](/figures/art_ad55f9ad1fdb4555a2e3f6a89259208b)

This results in "intermittent" sensing data—speech signals riddled with "null segments" where the device was powered off. Standard speech processing algorithms fail miserably on this incomplete data. While previous research addressed missing data, they often relied on simulated, arbitrary masking patterns. This paper tackles the specific, hardware-driven patterns of data loss found in real-world self-powered devices.

Methods

The proposed ISR system follows a three-stage pipeline to bridge the gaps in intermittent speech (see Figure 1):

  1. Null Segment Interpolation: The system first identifies gaps using timestamps. Instead of interpolating in the time domain, it performs linear interpolation in the frequency domain (STFT). The authors argue that speech harmonics are more observable and reconstructible in the frequency domain than in raw waveforms.
  2. DL-based Enhancement: A Complex U-Net architecture is used for refinement. Unlike standard models that only map magnitude, the complex-valued layers preserve crucial phase information. Crucially, the model is trained using a Perceptual Loss (feature loss) derived from a self-supervised encoder, which focuses on phonetic and speaker-representative information rather than just pixel-perfect signal matching.
  3. Combination: To ensure the recovery process doesn't accidentally degrade the parts of the signal that were successfully recorded, the system performs a final merge. It keeps the original recorded segments as-is and only uses the DL-enhanced output to fill the null segments.

![Fig.

  1. System architecture of the proposed ISR system.](/figures/art_38c48c193a3647dfa577488a12517dc8)

Datasets

The researchers evaluated their system using the VCTK-DEMAND corpus, a standard 16kHz speech dataset. To simulate real-world conditions:

  • Training: 11,572 utterances were corrupted using 13 different energy source levels (1.5 to 5.5 mW).
  • Testing: 824 unique utterances were corrupted by 4 different energy source levels (2.0 to 5.0 mW) that were not seen during training.
  • Evaluation Metrics: The team used PESQ (quality), STOI (intelligibility), and WER (accuracy via a pre-trained Google ASR system).

Key Insights and Ideas

  • Frequency Domain is King for Gaps: Linear interpolation in the waveform domain is "empirically undesirable" because it doesn't capture the harmonic structure of speech. Moving to the frequency domain allows for a much better initialization of the missing components.
  • The MSE Trap: The authors found that training with standard Mean Squared Error (MSE) loss actually deteriorated speech quality and intelligibility in some cases. MSE is not strongly correlated with how humans perceive speech. Switching to Perceptual Loss was the "aha moment" that unlocked significant gains in ASR accuracy.
  • Initialization Matters: Deep learning models for speech recovery are difficult to train from scratch on signals with large gaps. The first-stage interpolation provides a necessary "starting point" that makes the subsequent DL enhancement effective.

Key Results

The quantitative improvements are striking, especially in low-energy scenarios where data loss is most severe:

  • Speech Quality (PESQ): Increased by up to 591.7% (improving from a nearly unintelligible 0.24 to 1.66).
  • Intelligibility (STOI): Increased by up to 80.5%.
  • Recognition Accuracy (WER): Improved by up to 52.6%.

As seen in Figure 3, the system successfully fills the black "voids" in the intermittent spectrogram, reconstructing a signal that closely resembles the original clean speech.

Waveforms and spectrograms of an example utterance: (a) and (b) clean; (c) and (d) intermittent; (e) and (f) recovered by the ISR+PL.

Limitations and Future Work

While the results are promising, the authors note a few areas for further exploration:

  • Energy Floor: While the system performs well, the improvement is less significant when the energy source is extremely weak, as the "room for improvement" is limited by the sheer amount of missing data.
  • Computational Overhead: The paper assumes the DL-based recovery happens on a remote server (like a PC or cloud instance) rather than the IoT device itself. Future work could look at optimizing these models for more resource-constrained edge environments.
  • Real-time Constraints: The current pipeline involves multiple stages; the latency of this process for real-time applications was not explicitly discussed.

Summary updated

Abstract

The incompleteness of speech inputs severely degrades the performance of all the related speech signal processing applications. Although many researches have been proposed to address this issue, they controlled the data missing conditions by simulation with self-defined masking lengths or sizes. Besides, the masking definitions are different among all these experimental settings. This paper presents a novel intermittent speech recovery (ISR) system for real-world self-powered intermittent devices. Three contributive stages: interpolation, enhancement, and combination are applied to the ISR system for speech reconstruction. The experimental results show that our recovery system increases speech quality by up to 591.7%, while increasing speech intelligibility by up to 80.5%. Most importantly, the proposed ISR system improves the WER scores by up to 52.6%. The promising results not only confirm the effectiveness of the reconstruction but also encourage the utilization of these battery-free wearable/IoT devices.
BibTeX
@inproceedings{lin2022intermittent,
  author = {Yu-Chen Lin and Tsun-An Hsieh and Kuo-Hsuan Hung and Cheng Yu and Harinath Garudadri and Yu Tsao and Tei-Wei Kuo},
  title = {{Intermittent Speech Recovery}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=OvpdvtuOOW},
}
Figure