Abstract
In-home IoT devices play a major role in healthcare systems as smart personal assistants. They usually come with a voice-enabled feature to add an extra level of usability and convenience to elderly, disabled people, and patients. In this paper, we propose an efficient and privacy-preserving voice-based search scheme to enhance the efficiency and the privacy of in-home healthcare applications. We consider an application scenario where patients use the devices to record and upload their voice to servers and the caregivers search the interested voices of their patient's based on the voice content, mood, tone and background sound. Our scheme preserves the richness and privacy of voice data and enables accurate and efficient voice-based search, while in current systems that use speech recognition the richness and privacy of voice data are compromised. Specifically, our scheme achieves the privacy by employing a homomorphic encryption; only encrypted voice data is uploaded to the server who is unable to access the original voice data. In addition, our scheme enables the server to selectively and accurately respond to caregiver's queries on the voice data based on voice's feature similarity. We evaluate our scheme through real experiments and show that our scheme even with privacy preservation can successfully match similar voice data at an average accuracy of 80.8%.
Introduction
As smart speakers and in-home IoT devices become ubiquitous, they are increasingly integrated into mobile health (mHealth) systems to assist elderly and disabled patients. These devices offer a hands-free interface that is more accessible than traditional screens. However, current implementations often rely on converting voice to text for storage and analysis. This conversion process results in a significant loss of "acoustic richness"—the non-verbal cues such as tone, pitch, and background noise that can provide vital diagnostic information about a patient's emotional state or physical environment.
Furthermore, privacy remains a paramount concern. Uploading raw audio to cloud servers exposes sensitive health data to potential breaches or unauthorized access by service providers. This paper introduces a system designed to maintain the privacy of voice data while allowing caregivers to perform complex searches based on the acoustic characteristics of the recordings.
System Architecture
The proposed framework involves a secure pipeline between the patient's home and the caregiver's interface, mediated by a cloud-based database server.
As illustrated in the system architecture, the process begins with the user's voice input at an in-home device. The device performs local processing to extract features and encrypt the data before it ever reaches the cloud. The database server acts as a semi-trusted entity that performs matching operations on encrypted data without ever gaining access to the underlying audio content. Finally, the caregiver can query the system to find specific types of recordings (e.g., instances where a patient sounded distressed) and receive decrypted results.
Methodology
Feature Extraction
To capture the nuances of voice without storing raw audio, the system utilizes Mel Frequency Cepstral Coefficients (MFCC). This algorithm creates a 36-dimensional feature matrix that acts as a unique acoustic fingerprint.
The extraction process follows a standard signal processing flow: the speech signal is pre-emphasized and framed, followed by a Fast Fourier Transform (FFT). The signal then passes through a Mel-frequency filter bank and cepstrum analysis to produce the final speech signal features. This ensures that characteristics like mood and tone are preserved in the metadata.
Privacy-Preserving Search via Homomorphic Encryption
The core innovation lies in the hybrid encryption approach. While the raw audio is secured using standard AES encryption, the MFCC features are protected using Homomorphic Encryption (HE). HE allows the server to compute the Euclidean distance between a caregiver's query and the stored patient records without decrypting either. This "blind" matching ensures that the server facilitates the search but never learns the content of the query or the results.
Experimental Evaluation
The researchers tested the system using a combination of machine-generated voices and real-world human recordings captured in various environments. The system achieved an overall average accuracy of 80.8%.
Specific performance highlights include:
- High-Fidelity Matching: 86.66% accuracy when matching the same word spoken with the same mood.
- Semantic Flexibility: 79.15% accuracy when matching the same word spoken with different moods, demonstrating the system's ability to distinguish between content and delivery.
- Environmental Awareness: 76.66% accuracy in identifying similar background noises, which can help caregivers understand the patient's context (e.g., identifying if a patient is in a noisy or quiet room).
Conclusion
This research demonstrates that it is possible to build a voice-based search system for mHealth that is both highly functional and strictly private. By moving feature extraction to the edge and utilizing homomorphic encryption in the cloud, the system preserves the diagnostic richness of voice data while protecting patient confidentiality. Future work will focus on refining the feature extraction process to improve accuracy beyond the 80.8% threshold and incorporating a broader range of vocal characteristics into the search parameters.
Summary updated