TL;DR
This paper introduces ERSAM, a novel neural architecture search framework designed to create highly efficient, real-time deep learning models for measuring social ambiance (specifically, counting concurrent speakers) directly on mobile devices. ERSAM achieves a remarkable balance between accuracy and hardware constraints, enabling privacy-preserving social ambiance analysis for applications like mental health tracking.
What Problem Does This Paper Address?
Social ambiance, particularly the number of people speaking concurrently, is a valuable indicator for various applications, including mental health monitoring. While deep neural networks (DNNs) can accurately measure this, deploying them on resource-constrained mobile devices presents a significant challenge. State-of-the-art DNNs are computationally intensive, exceeding the power and latency budgets of typical smartphones. Furthermore, collecting and labeling large datasets for training these models is difficult due to privacy concerns and the effort involved, especially in clinical settings. This gap between the need for on-device, real-time social ambiance measurement and the limitations of current DNN technology is what ERSAM aims to solve.
Methods
ERSAM is a comprehensive framework that combines two key innovations:
- Hardware-Aware Neural Architecture Search (HW-NAS) for SAM: Instead of using generic search spaces, ERSAM develops a specialized search space tailored for social ambiance measurement. This is informed by analyzing the latency and energy consumption of operators in existing speech models (like wav2vec) on mobile hardware. The goal is to identify efficient network architectures that are optimized for the specific computational characteristics of SAM tasks on edge devices.
- Efficient Knowledge Distillation: To further boost the accuracy of compact models without incurring excessive training costs, ERSAM incorporates an efficient knowledge distillation scheme. This method leverages a larger, pre-trained "teacher" model to guide the training of a smaller "student" model. The key innovation here is that the distillation process is only activated when the student model exhibits high uncertainty in its predictions. This selective distillation significantly reduces the computational overhead compared to traditional knowledge distillation, which often requires constant interaction with the teacher model.
ERSAM integrates these enablers into a unified search and training process, aiming to automatically discover DNNs that meet stringent real-time, energy, and data efficiency requirements.
Datasets
- LibriSpeech-SAM: A synthesized dataset created from the LibriSpeech corpus. It consists of 5-second audio segments with varying numbers of concurrent speakers (0 to ≥5). This dataset was generated to address the lack of realistic, labeled social ambiance data. It includes 8 hours for training/validation and 2 hours for testing, with balanced class distributions.
- TUT Acoustic Scenes dataset: Used to provide non-speech samples (k=0) for training.
- LibriAdapt: Used for simulating domain shifts caused by different recording devices.
Key Insights and Ideas
- Hardware-Specific Search is Crucial: The paper highlights that the computational cost of DNN operators varies significantly across different hardware. A generic search space is insufficient; a hardware-aware approach tailored to the target device and task (SAM) is essential for achieving true efficiency.
- Uncertainty-Driven Distillation is Efficient: The insight that larger models excel when smaller models are uncertain is powerful. By only distilling knowledge when the student model is "unsure," ERSAM dramatically reduces the computational burden of knowledge distillation, making it practical for on-device training scenarios.
- Balancing Accuracy and Efficiency is Achievable: The results demonstrate that it's possible to achieve high accuracy comparable to larger models while meeting strict energy and latency constraints, a significant step towards ubiquitous on-device AI.
Key Results
ERSAM successfully searched for and delivered DNNs that meet the following criteria on a Pixel 3 phone:
- Latency: 0.05 seconds for a 5-second audio segment (well within the <5 second requirement).
- Energy Consumption: 40 mW for 12 hours of active use (significantly lower than the <700 mW 12h requirement).
- Accuracy: 14.3% error rate on the LibriSpeech-SAM dataset. This is a notable improvement over the previous state-of-the-art (CountNet) which had a 16.7% error rate.
- Training Data: Achieved these results with only 8 hours of training data, addressing the data scarcity challenge.
Limitations and Future Work
- Dataset Synthesis: While LibriSpeech-SAM is a good proxy, the authors acknowledge that real-world, clinically collected datasets would be ideal for further validation.
- Generalization: While the paper shows generalization to noisy conditions and different devices (Tab. 4), further exploration of robustness across a wider range of real-world acoustic environments could be beneficial.
- Future Work: The authors suggest that ERSAM can be extended to other on-device audio analysis tasks that require similar efficiency and privacy considerations. The framework's ability to automatically discover efficient models opens doors for broader applications in human-centric IoT and edge AI.
Summary updated