Privacy-Preserving Deep Speaker Separation for Smartphone-Based Passive Speech Assessment

Apiwat Ditthapron, Emmanuel Agu, Adam C. LammertIEEE Open Journal of Engineering in Medicine and Biology 2021 · DOI

Abstract

Goal: Smartphones can be used to passively assess and monitor patients' speech impairments caused by ailments such as Parkinson's disease, Traumatic Brain Injury (TBI), Post-Traumatic Stress Disorder (PTSD) and neurodegenerative diseases such as Alzheimer's disease and dementia. However, passive audio recordings in natural settings often capture the speech of non-target speakers (cross-talk). Consequently, speaker separation, which identifies the target speakers' speech in audio recordings with two or more speakers' voices, is a crucial pre-processing step in such scenarios. Prior speech separation methods analyzed raw audio. However, in order to preserve speaker privacy, passively recorded smartphone audio and machine learning-based speech assessment are often performed on derived speech features such as Mel-Frequency Cepstral Coefficients (MFCCs). In this paper, we propose a novel Deep MFCC bAsed SpeaKer Separation (Deep-MASKS). Methods: Deep-MASKS uses an autoencoder to reconstruct MFCC components of an individual's speech from an i-vector, x-vector or d-vector representation of their speech learned during the enrollment period. Deep-MASKS utilizes a Deep Neural Network (DNN) for MFCC signal reconstructions, which yields a more accurate, higher-order function compared to prior work that utilized a mask. Unlike prior work that operates on utterances, Deep-MASKS operates on continuous audio recordings. Results: Deep-MASKS outperforms baselines, reducing the Mean Squared Error (MSE) of MFCC reconstruction by up to 44% and the number of additional bits required to represent clean speech entropy by 36%.

Introduction

Passive speech assessment—using smartphone microphones to monitor for signs of Parkinson’s, Alzheimer’s, or PTSD—is a burgeoning field. However, real-world environments are messy. "Cross-talk" (when a non-target speaker is captured by the microphone) can lead to erroneous health assessments.

While speaker separation is a known solution, most existing methods operate on raw audio. In clinical and mobile settings, storing raw audio is a major privacy risk. To mitigate this, many systems immediately convert audio into Mel-Frequency Cepstral Coefficients (MFCCs)—a compact, "featurized" representation that is harder to reconstruct into intelligible speech. The challenge is that once you are in the MFCC domain, traditional waveform-based separation tools no longer work. This paper introduces a way to separate speakers after the audio has been converted to MFCCs, maintaining the privacy-preserving benefits of the feature-only pipeline.

Proposed Methodology

The authors propose Deep-MASKS (Deep MFCC bAsed SpeaKer Separation). Unlike traditional methods that use a "mask" (a filter that suppresses certain frequencies), Deep-MASKS uses a Deep Neural Network (DNN) to perform a full signal reconstruction. This approach is specifically designed for continuous audio recordings rather than isolated utterances, making it suitable for the "always-on" nature of passive smartphone monitoring.

Comparison of the traditional speech processing pipeline versus the proposed Deep-MASKS pipeline.

The architecture consists of three primary neural components:

  1. Encoder Network: A pre-trained bi-directional LSTM (bi-LSTM) autoencoder that transforms input MFCCs into a compact bottleneck feature.
  2. Speaker Matching Network: This layer takes the bottleneck feature and concatenates it with a pre-learned "speaker representation" (an embedding like a d-vector or x-vector) of the phone's owner. It performs similarity matching to identify which parts of the signal belong to the target user.
  3. MFCC Refinement Network: A final stage that reconstructs the clean MFCCs of the target speaker, effectively "subtracting" the background voices.
Detailed architecture of the Deep-MASKS model showing the integration of speaker representations.

Experimental Setup

  • AMI Meeting Corpus: Used for primary training and evaluation, containing 100 hours of natural, multi-speaker meetings. Smartphone dynamics were simulated by combining audio from different headsets to create realistic overlapped speech.
  • VoxCeleb: A large-scale speaker identification dataset used to pre-train the speaker representation models (i-vector, x-vector, and d-vector) used by the matching network.

Key Results and Insights

  • Accuracy: Deep-MASKS reduced the Mean Squared Error (MSE) of MFCC reconstruction by up to 44% compared to the Deep Attractor Network (DAN) baseline.
  • Information Efficiency: It reduced the number of additional bits required to represent clean speech entropy by 36%.
  • Embedding Flexibility: While the system is "embedding-agnostic," the study found that d-vectors (deep learning-based embeddings) provided the best performance for this specific task, consistently outperforming i-vectors and x-vectors in scenarios with up to four speakers.
  • Reconstruction over Masking: By using a DNN for reconstruction rather than linear masking, Deep-MASKS can model higher-order, non-linear relationships in the speech data, leading to significantly cleaner output.

Limitations and Future Work

While the system successfully removes cross-talk, it can introduce minor artifacts, particularly during periods of silence or non-speech, as it currently lacks a built-in Voice Activity Detection (VAD) mechanism. Additionally, while optimized for smartphones, future research is needed to benchmark battery consumption and real-time processing latency on mobile hardware.

Finally, the authors suggest that future work should validate this pipeline on actual clinical populations (e.g., patients with Parkinson's) to ensure the reconstructed MFCCs retain the specific pathological markers needed for accurate diagnosis.

Summary updated

Abstract

<i>Goal:</i> Smartphones can be used to passively assess and monitor patients' speech impairments caused by ailments such as Parkinson's disease, Traumatic Brain Injury (TBI), Post-Traumatic Stress Disorder (PTSD) and neurodegenerative diseases such as Alzheimer's disease and dementia. However, passive audio recordings in natural settings often capture the speech of non-target speakers (cross-talk). Consequently, speaker separation, which identifies the target speakers' speech in audio recordings with two or more speakers' voices, is a crucial pre-processing step in such scenarios. Prior speech separation methods analyzed raw audio. However, in order to preserve speaker privacy, passively recorded smartphone audio and machine learning-based speech assessment are often performed on derived speech features such as Mel-Frequency Cepstral Coefficients (MFCCs). In this paper, we propose a novel Deep MFCC bAsed SpeaKer Separation (Deep-MASKS). <i>Methods:</i> Deep-MASKS uses an autoencoder to reconstruct MFCC components of an individual's speech from an i-vector, x-vector or d-vector representation of their speech learned during the enrollment period. Deep-MASKS utilizes a Deep Neural Network (DNN) for MFCC signal reconstructions, which yields a more accurate, higher-order function compared to prior work that utilized a mask. Unlike prior work that operates on utterances, Deep-MASKS operates on continuous audio recordings. <i>Results:</i> Deep-MASKS outperforms baselines, reducing the Mean Squared Error (MSE) of MFCC reconstruction by up to 44% and the number of additional bits required to represent clean speech entropy by 36%.
BibTeX
@article{ditthapron2021privacy,
  author = {Apiwat Ditthapron and Emmanuel Agu and Adam C. Lammert},
  title = {{Privacy-Preserving Deep Speaker Separation for Smartphone-Based Passive Speech Assessment}},
  journal = {IEEE Open Journal of Engineering in Medicine and Biology},
  year = {2021},
  doi = {10.1109/ojemb.2021.3063994},
}
Figure