Abstract
For social robots to function effectively in real-world environments, they must be able to process human speech while simultaneously suppressing both internal motor sounds (ego-noise) and external ambient sounds (environmental noise). This paper presents a multichannel framework utilizing a Variational Autoencoder (VAE) for speech modeling combined with a Non-negative Matrix Factorization (NMF) approach for noise reduction. By employing a "partially adaptive" strategy—where ego-noise parameters are pre-learned and fixed while environmental noise parameters remain fully adaptive—the proposed system significantly improves speech quality and recognition accuracy over traditional fully adaptive or fixed baselines.
Introduction
Social robots are increasingly deployed in complex acoustic environments where they must interact naturally with humans. However, the proximity of microphones to the robot's own actuators creates a significant challenge: ego-noise. This noise is often loud, non-stationary, and highly complex. While existing research has addressed ego-noise (which is predictable based on robot movement) and environmental noise (which is unpredictable) as separate problems, real-world scenarios require a joint reduction strategy.
Standard fully adaptive models often struggle to distinguish complex motor sounds from human speech, leading to signal distortion. Conversely, fixed models cannot account for the dynamic nature of external environments. This paper proposes a hybrid approach that exploits the known spatial and spectral "fingerprints" of the robot's hardware while maintaining the flexibility to adapt to unknown surroundings.
Methodology
The researchers developed a multichannel framework based on a VAE-NMF architecture. This approach integrates deep generative modeling for speech with classical signal processing for noise decomposition.
1. Speech Modeling (VAE)
A Variational Autoencoder is utilized to provide a probabilistic prior for clean speech. Pre-trained on the TIMIT dataset, the VAE allows the system to recognize the underlying structure of human voices, helping the filter distinguish speech from broadband noise.
2. Ego-Noise Modeling (Fixed NMF)
Because a robot’s motor noise is tied to its physical structure, its spatial and spectral characteristics are relatively consistent for specific movements. The authors pre-train a Multichannel NMF model on isolated recordings of the robot's joints. During live operation, the spectral dictionary and spatial covariance matrices are kept fixed, while only the temporal activations (the timing of the noise) are updated. This "protects" the speech signal from being erroneously suppressed by the adaptive components of the model.
3. Environmental Noise Modeling (Adaptive NMF)
To handle unpredictable external sounds (e.g., background chatter or machinery), a second NMF model is employed. Unlike the ego-noise model, all parameters—spatial, spectral, and temporal—are fully adaptive and estimated in real-time.
4. Inference and Extraction
The system employs a Monte Carlo Expectation Maximization (MCEM) algorithm to estimate the latent variables of the joint model. The final clean speech signal is extracted using a multichannel Wiener filter, which leverages both the spectral and spatial cues provided by the 4-microphone array.
Experimental Design
Datasets and Setup
- Speech: Clean utterances from the TIMIT test set.
- Ego-Noise: Custom recordings from a NAO H25 humanoid robot performing specific arm movements in a crouching posture.
- Environmental Noise: Samples from the DEMAND database, representing diverse real-world ambient settings.
- Hardware: A 4-microphone array mounted on the NAO robot within a varechoic chamber. Loudspeakers were used to simulate human speakers and external noise sources at various angles.
Evaluation Metrics
The "Partial" adaptive scheme was benchmarked against "Fully Adaptive" (where all noise parameters are learned on the fly) and "Fully Fixed" (where all noise parameters are pre-trained) baselines using three primary metrics: Signal-to-Distortion Ratio (Si-SDR), Perceptual Objective Analysis of Speech Quality (POLQA), and Word Error Rate (WER).
Results and Discussion
The experimental results demonstrate that the partially adaptive framework offers a superior balance between noise suppression and signal preservation.
- Signal Quality: In joint noise scenarios (Ego + Environmental), the Partial scheme consistently outperformed baselines in Si-SDR. As the NMF dictionary size increased, the Partial scheme maintained high performance, whereas the Fully Adaptive model began to overfit and distort the speech signal.
- Perceptual Clarity: POLQA scores indicated that the proposed method produced more natural and intelligible speech compared to the robotic or muffled outputs of the baseline models.
- Speech Recognition: Using the Quartznet model for automated speech recognition, the Partial scheme achieved a Word Error Rate (WER) of approximately 0.65–0.70. In contrast, the Fully Adaptive model’s performance degraded significantly in the presence of joint noise, with WER exceeding 0.85.
These results suggest that "freezing" the known characteristics of the robot's own noise allows the adaptive portion of the algorithm to focus more effectively on the unknown environmental variables.
Conclusion and Future Work
The partially adaptive VAE-NMF framework represents a significant step forward in robotic audition. By combining pre-learned hardware signatures with real-time environmental adaptation, robots can better understand human commands in noisy, real-world settings.
However, limitations remain. The current ego-noise model is movement-specific; if the robot performs a novel motion, the fixed parameters may become inaccurate. Future research will explore the integration of internal motor data (such as joint velocity or electrical current) directly into the NMF activations to improve timing. Additionally, optimizing the MCEM inference process for low-power, real-time deployment on robot hardware remains a priority.
Summary updated