Abstract: This paper presents a Speech Enhancement (SE) technique based on multi-objective learning convolutional neural network to improve the overall quality of speech perceived by Hearing Aid (HA) users. The proposed method is implemented on a smartphone as an application that performs real-time SE. This arrangement works as an assistive tool to HA. A multi-objective learning architecture including primary and secondary features uses a mapping-based convolutional neural network (CNN) model to remove noise from a noisy speech spectrum. The algorithm is computationally fast and has a low processing delay which enables it to operate seamlessly on a smartphone. The steps and the detailed analysis of real-time implementation are discussed. The proposed method is compared with existing conventional and neural network-based SE techniques through speech quality and intelligibility metrics in various noisy speech conditions. The key contribution of this paper includes the realization of CNN SE model on a smartphone processor that works seamlessly with HA. The experimental results demonstrate significant improvements over the state-of-the-art techniques and reflect the usability of the developed SE application in noisy environments.
Introduction
Hearing Aid Devices (HADs) are essential tools for individuals with hearing impairments, yet they face a significant challenge: background noise. While effective in quiet settings, HADs often struggle in "cocktail party" environments—such as crowded restaurants or busy streets—where multiple sound sources compete. The primary constraint is hardware; the small form factor of HADs limits battery life and computational power, making it impossible to run complex deep-learning models for high-quality speech enhancement (SE). This paper proposes a novel solution: using the smartphone as an external "brain" to perform real-time SE, offloading heavy computation to its more powerful processor.
Proposed Methodology
The core of the proposed system is a mapping-based Convolutional Neural Network (CNN) designed for both efficiency and perceptual accuracy.
Multi-Objective Learning
Unlike traditional models that focus solely on reconstructing clean audio, this system employs multi-objective learning. The network is trained to predict two distinct features: the clean Log Power Spectra (LPS) as the primary target and the Log Mel-filterbank energy (MFSC) as a secondary target. This secondary feature mimics human auditory perception, guiding the model to prioritize phonetically significant components of the speech signal.
CNN Architecture and Efficiency
The CNN architecture is intentionally shallow to minimize latency. It consists of two convolutional layers followed by a fully connected layer. To maintain real-time performance, the researchers implemented several optimizations:
- Strided Convolutions: A stride of 3 is used instead of max-pooling to reduce data dimensionality without excessive computational overhead.
- Zero Padding: This ensures the output temporal length matches the input, facilitating seamless real-time reconstruction.
Real-Time Pipeline
The smartphone application captures audio at 48 kHz, which is then downsampled to 8 kHz for processing. A 32ms frame is processed by the CNN, and the enhanced audio is delivered to the hearing aid via low-latency Bluetooth or a wired connection.
Experimental Setup
The model was trained on a comprehensive dataset totaling approximately 33 hours of noisy speech. This included speech from the HINT, TIMIT, and LibriSpeech corpora, combined with noise from the DCASE 2017 challenge and 30 realistic noise types recorded specifically on smartphones to account for real-world microphone characteristics.
Results and Discussion
The proposed CNN-based SE system was evaluated against traditional statistical methods (e.g., Log-MMSE) and standard Deep Neural Networks (DNNs).
Performance Metrics
- Objective Quality: The CNN consistently outperformed baseline methods in Perceptual Evaluation of Speech Quality (PESQ) and Log Spectral Distance (LSD) across various noise conditions, including machinery, babble, and traffic.
- Subjective Evaluation: In Mean Opinion Score (MOS) tests, human listeners preferred the proposed method over all other neural and conventional techniques.
- Latency and Resource Usage: On an iPhone X, the processing time for a 32ms frame was approximately 14ms. This low latency allows for continuous operation without audio lag. The application utilized roughly 29% of the CPU and 308MB of RAM, supporting about 5 hours of continuous use.
Key Insights
The success of this approach lies in leveraging the smartphone as a co-processor. Modern mobile CPUs are highly capable of running sophisticated SE models that are beyond the reach of internal HAD hardware. Furthermore, using CNNs provides translational invariance, allowing the model to recognize speech patterns even with slight shifts in pitch or timing. The multi-objective training acts as a regularizer, ensuring the learned representations align better with human hearing.
Conclusion and Future Work
This research demonstrates the feasibility of a real-time, smartphone-based SE system for hearing aid users. While the current implementation provides significant improvements in speech quality and intelligibility, future work will focus on optimizing battery consumption, perhaps by utilizing hardware-specific neural engines. Additionally, addressing end-to-end Bluetooth latency and developing more universal noise suppression models remain key areas for further development.
Summary updated