ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement

Chaojian Li, Wenwan Chen, Jiayi Yuan, Yingyan Lin, Ashutosh Sabharwal

TL;DR

This paper introduces ERSAM, a novel neural architecture search framework designed to create highly efficient, real-time deep learning models for measuring social ambiance (specifically, counting concurrent speakers) directly on mobile devices. ERSAM achieves a remarkable balance between accuracy and hardware constraints, enabling privacy-preserving social ambiance analysis for applications like mental health tracking.

What Problem Does This Paper Address?

Social ambiance, particularly the number of people speaking concurrently, is a valuable indicator for various applications, including mental health monitoring. While deep neural networks (DNNs) can accurately measure this, deploying them on resource-constrained mobile devices presents a significant challenge. State-of-the-art DNNs are computationally intensive, exceeding the power and latency budgets of typical smartphones. Furthermore, collecting and labeling large datasets for training these models is difficult due to privacy concerns and the effort involved, especially in clinical settings. This gap between the need for on-device, real-time social ambiance measurement and the limitations of current DNN technology is what ERSAM aims to solve.

Methods

ERSAM is a comprehensive framework that combines two key innovations:

  1. Hardware-Aware Neural Architecture Search (HW-NAS) for SAM: Instead of using generic search spaces, ERSAM develops a specialized search space tailored for social ambiance measurement. This is informed by analyzing the latency and energy consumption of operators in existing speech models (like wav2vec) on mobile hardware. The goal is to identify efficient network architectures that are optimized for the specific computational characteristics of SAM tasks on edge devices.
  2. Efficient Knowledge Distillation: To further boost the accuracy of compact models without incurring excessive training costs, ERSAM incorporates an efficient knowledge distillation scheme. This method leverages a larger, pre-trained "teacher" model to guide the training of a smaller "student" model. The key innovation here is that the distillation process is only activated when the student model exhibits high uncertainty in its predictions. This selective distillation significantly reduces the computational overhead compared to traditional knowledge distillation, which often requires constant interaction with the teacher model.

ERSAM integrates these enablers into a unified search and training process, aiming to automatically discover DNNs that meet stringent real-time, energy, and data efficiency requirements.

Datasets

  • LibriSpeech-SAM: A synthesized dataset created from the LibriSpeech corpus. It consists of 5-second audio segments with varying numbers of concurrent speakers (0 to ≥5). This dataset was generated to address the lack of realistic, labeled social ambiance data. It includes 8 hours for training/validation and 2 hours for testing, with balanced class distributions.
  • TUT Acoustic Scenes dataset: Used to provide non-speech samples (k=0) for training.
  • LibriAdapt: Used for simulating domain shifts caused by different recording devices.

Key Insights and Ideas

  • Hardware-Specific Search is Crucial: The paper highlights that the computational cost of DNN operators varies significantly across different hardware. A generic search space is insufficient; a hardware-aware approach tailored to the target device and task (SAM) is essential for achieving true efficiency.
  • Uncertainty-Driven Distillation is Efficient: The insight that larger models excel when smaller models are uncertain is powerful. By only distilling knowledge when the student model is "unsure," ERSAM dramatically reduces the computational burden of knowledge distillation, making it practical for on-device training scenarios.
  • Balancing Accuracy and Efficiency is Achievable: The results demonstrate that it's possible to achieve high accuracy comparable to larger models while meeting strict energy and latency constraints, a significant step towards ubiquitous on-device AI.

Key Results

ERSAM successfully searched for and delivered DNNs that meet the following criteria on a Pixel 3 phone:

  • Latency: 0.05 seconds for a 5-second audio segment (well within the <5 second requirement).
  • Energy Consumption: 40 mW for 12 hours of active use (significantly lower than the <700 mW 12h requirement).
  • Accuracy: 14.3% error rate on the LibriSpeech-SAM dataset. This is a notable improvement over the previous state-of-the-art (CountNet) which had a 16.7% error rate.
  • Training Data: Achieved these results with only 8 hours of training data, addressing the data scarcity challenge.

Limitations and Future Work

  • Dataset Synthesis: While LibriSpeech-SAM is a good proxy, the authors acknowledge that real-world, clinically collected datasets would be ideal for further validation.
  • Generalization: While the paper shows generalization to noisy conditions and different devices (Tab. 4), further exploration of robustness across a wider range of real-world acoustic environments could be beneficial.
  • Future Work: The authors suggest that ERSAM can be extended to other on-device audio analysis tasks that require similar efficiency and privacy considerations. The framework's ability to automatically discover efficient models opens doors for broader applications in human-centric IoT and edge AI.

Summary updated

Abstract

Social ambiance describes the context in which social interactions happen, and can be measured using speech audio by counting the number of concurrent speakers. This measurement has enabled various mental health tracking and human-centric IoT applications. While on-device Socal Ambiance Measure (SAM) is highly desirable to ensure user privacy and thus facilitate wide adoption of the aforementioned applications, the required computational complexity of state-of-the-art deep neural networks (DNNs) powered SAM solutions stands at odds with the often constrained resources on mobile devices. Furthermore, only limited labeled data is available or practical when it comes to SAM under clinical settings due to various privacy constraints and the required human effort, further challenging the achievable accuracy of on-device SAM solutions. To this end, we propose a dedicated neural architecture search framework for Energy-efficient and Real-time SAM (ERSAM). Specifically, our ERSAM framework can automatically search for DNNs that push forward the achievable accuracy vs. hardware efficiency frontier of mobile SAM solutions. For example, ERSAM-delivered DNNs only consume 40 mW x 12 h energy and 0.05 seconds processing latency for a 5 seconds audio segment on a Pixel 3 phone, while only achieving an error rate of 14.3% on a social ambiance dataset generated by LibriSpeech. We can expect that our ERSAM framework can pave the way for ubiquitous on-device SAM solutions which are in growing demand.
BibTeX
@inproceedings{li2023ersam,
  author = {Chaojian Li and Wenwan Chen and Jiayi Yuan and Yingyan Lin and Ashutosh Sabharwal},
  title = {{ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=8ySmOwbFsG},
}
Figure