Challenges in the Processing of Audio Channels for Ambient Assisted Living

Michel Vacher, François Portet, Anthony Fleury, Norbert Noury

TL;DR

This paper explores the challenges of using audio processing for Ambient Assisted Living (AAL) systems, focusing on recognizing distress calls and classifying daily living sounds. The authors introduce their AuditHIS system and present experimental results highlighting the difficulties of real-world AAL environments, such as noise and reverberation, and the need for speaker-adapted ASR for the elderly.

What Problem Does This Paper Address?

The increasing aging population and the desire for independent living present a significant challenge for healthcare and social support systems. Ambient Assisted Living (AAL) aims to address this by creating "smart homes" that can anticipate inhabitants' needs, enhance comfort and safety, and facilitate connections to the outside world. A key aspect of AAL is enabling intuitive interaction with these smart environments, especially for individuals with reduced autonomy. While visual and motion sensors are common, audio channels offer a natural and powerful interface, particularly for vocal commands and detecting distress situations. However, applying audio processing in real-world home environments is fraught with difficulties, including background noise, reverberation, and the need to adapt to diverse users, especially the elderly, whose speech patterns can differ significantly. This paper investigates these challenges by evaluating an audio processing system in a realistic smart home setting.

Methods

The core of the paper's technical contribution is the AuditHIS system, a real-time, multi-source audio analysis system designed for smart homes. AuditHIS operates in several independent threads:

  • Acquisition and First Analysis: Captures audio from multiple microphones (8 in their setup) at a 16 kHz sampling rate and calculates the Signal-to-Noise Ratio (SNR) for each channel.
  • Detection: Identifies the beginning and end of audio events using an adaptive threshold based on background noise estimation.
  • Discrimination: Uses Gaussian Mixture Models (GMMs) to distinguish between everyday sounds and speech.
  • Classification:
    • Sound Classifier: Another GMM classifies everyday sounds into predefined categories.
  • Autonomous Speech Recognizer (RAPHAEL): For speech events, RAPHAEL, an independent application, performs speech recognition. It utilizes speaker-independent acoustic models trained on large French corpora and a language model based on n-grams, specifically designed for keyword detection in distress situations.

The system is designed to process audio events on the fly and format them into XML output for further analysis.

Datasets

The paper describes two primary experimental setups and the resulting datasets:

  • Distress Call Recognition Experiment:
    • Participants: Ten native French speakers.
  • Data: Participants uttered 45 sentences each (20 distress, 10 normal, and 3 phone conversations of 5 sentences each).
  • Resulting Corpus: 197 distress keyword sentences (nDS) and 232 normal sentences (nNS), totaling 429 sentences. This corpus was manually indexed due to participants not strictly following instructions.
  • Daily Living Sounds and Speech Experiment:
    • Participants: Fifteen healthy participants (including 6 women).
  • Data: Participants performed seven Activities of Daily Living (ADLs): Sleeping, Resting, Dressing/Undressing, Feeding, Eliminating, Hygiene, and Communicating.
  • Resulting Corpus: 1886 individual sounds and 669 sentences were collected and manually annotated.

The paper also mentions the use of a "Normal/Distress speech corpus" for training the discrimination module and an "everyday life sound corpus" for the sound classifier, though specific details on these training corpora are limited.

Key Insights and Ideas

  • Real-world audio is messy: The experiments starkly reveal that the controlled laboratory environment is vastly different from a real smart home. Low SNRs (around 11 dB in the smart home vs. 27 dB in the lab) and significant reverberation are major hurdles.
  • Speaker adaptation is crucial for ASR: The performance of the speech recognition system (RAPHAEL) varied significantly across speakers, with some achieving low error rates while others struggled. This highlights the need to adapt ASR models to individual users, especially the elderly, whose speech characteristics can change with age.
  • Distress detection is challenging without full understanding: The goal of recognizing distress keywords without necessarily understanding the entire conversation is a pragmatic approach. However, even this is difficult when faced with variations in speech intensity (e.g., acting) or background noise.
  • Sound/Speech discrimination is a critical bottleneck: The high confusion rate between everyday sounds and speech (e.g., dishes vs. speech, falls vs. screams) significantly impacts the performance of both the sound classifier and the ASR. This suggests that the initial discrimination step is a major area for improvement.
  • The "unknown" and "mixed" sound categories are substantial: A significant portion of recorded sounds were either unclassifiable or a mix of multiple sounds. This indicates that robust systems need to handle ambiguity and have a mechanism for rejecting or flagging uncertain events.

Key Results

  • Distress Call Recognition:
    • The False Alarm Rate (FAR) was generally low (around 4% overall), indicating that the system rarely flagged normal sentences as distress.
  • However, the Missed Alarm Rate (MAR) and Global Error Rate (GER) showed high variability across speakers (from ~5% to over 50%).
  • Specific issues were noted, such as the French pronoun "je" being misrecognized due to variations in intensity and the French phoneme [h] not existing, causing problems with English-pronounced distress words.
  • Daily Living Sounds and Speech:
    • The sound/speech discrimination stage showed a high confusion rate, with everyday sounds being misclassified as speech 23% of the time and speech as everyday sounds 25% of the time.
  • This confusion was attributed to insufficient training data for certain sound classes and the inherent ambiguity between some sounds and speech (e.g., screams).

Limitations and Future Work

Limitations: * Noisy and reverberant environment: The smart home environment presented significant challenges due to noise and reverberation, impacting audio quality. * Limited training data: The paper explicitly states that the poor performance of classifiers was often due to small training sets and the lack of examples for unknown or unexpected sound classes. * Speaker variability: The ASR system's performance was highly dependent on the speaker, indicating a need for adaptation. * Ambiguity in sound classification: A significant percentage of sounds were mixed or unclassifiable, posing a challenge for categorization.

Future Work: * Improve audio acquisition and processing: Address issues like microphone placement, acoustic properties of the flat, and background noise. * Enhance sound/speech discrimination: Extend training datasets with more examples, variations, and a "reject class" for unknown sounds. * Develop speaker-adapted ASR: Focus on adapting speech recognition models to the aging population, potentially through on-line adaptation of recognized words. * Explore hierarchical classification: Investigate more sophisticated classification methods that can handle ambiguity and structured data. * Blind source separation: Develop techniques to disentangle simultaneous audio events from multiple microphones. * Multi-modal fusion: While not the primary focus, the paper implies that integrating audio with other sensor data could improve overall system performance. * Recording seniors' voices: The authors are actively recording voices of seniors to adapt their ASR system for this specific demographic.

Summary updated

Abstract

One of the greatest challenges in Ambient Assisted Living is to design health smart homes that could be able to anticipate the needs of its inhabitant while maintaining their comfort and their safety with an adaptation of the house environment and a facilitation of the connections to the outside world. The most likely to benefit from these smart homes are people in loss of autonomy such as the disabled people or the elderly with cognitive deficiencies. But it becomes essential to ease the interactions with the smart home through dedicated interfaces, in particular, thanks to systems reactive to vocal orders. Audio recognition is also a promising way to ensure more safety by contributing to detection of distress situations. This paper presents the stakes and the challenges of this domain based on some experiments carried out concerning distress call recognition and sound classification at home. I.
BibTeX
@misc{vacher2010challenges,
  author = {Michel Vacher and François Portet and Anthony Fleury and Norbert Noury},
  title = {{Challenges in the Processing of Audio Channels for Ambient Assisted Living}},
  year = {2010},
}

Loading related content...

Figure