ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement

Chaojian Li, Wenwan Chen, Jiayi Yuan, Yingyan Celine Lin, Ashutosh Sabharwal

Counting the number of people speaking at once—Social Ambiance Measurement (SAM)—is a powerful tool for tracking mental health and building smarter IoT environments. However, doing this privately and continuously on a mobile device is a massive technical hurdle. A new framework called ERSAM (Energy-efficient and Real-time SAM) aims to bridge this gap using specialized Neural Architecture Search (NAS).

TL;DR

ERSAM uses hardware-aware neural architecture search and a novel, uncertainty-driven knowledge distillation scheme to design speaker-counting models specifically for mobile devices. It achieves state-of-the-art accuracy while operating well within the strict power and latency budgets required for real-time, privacy-preserving mental health monitoring.

What Problem Does This Paper Address?

While deep neural networks (DNNs) are great at counting speakers, they are usually "heavyweight." State-of-the-art models like wav2vec 2.0 are designed for high-performance servers, not smartphones. For example, running a full wav2vec model on a Pixel 3 consumes roughly three times the power the device can actually afford for continuous background monitoring.

Furthermore, training these models typically requires massive amounts of labeled data. In clinical settings, collecting and labeling hundreds of hours of private audio is both a privacy nightmare and prohibitively expensive. ERSAM tackles this "triple threat": the need for real-time latency (processing 5s of audio in <5s), low energy consumption (staying under 700mW for all-day use), and high accuracy with limited training data (less than 8 hours).

Methods

The ERSAM framework introduces two primary "enablers" to solve these problems:

  1. Hardware-Aware Search Space (Enabler 1): The authors first profiled existing models to find hardware bottlenecks. They discovered that specific combinations—like large kernel sizes paired with long input sequences—accounted for over 35% of total latency on a Pixel 3 (see Figure 4 below). They designed a custom search space for their NAS engine that intentionally avoids these "bottleneck" configurations while still allowing for high-capacity models.
Fig. 3: Latency of different operators in wav2vec [32] profiled on a Pixel 3 [4] phone.
  1. Efficient Knowledge Distillation (Enabler 2): Knowledge distillation (KD) usually involves a small "student" model learning from a giant "teacher" model. Standard KD is slow because you have to query the teacher for every single training sample. ERSAM introduces an uncertainty-based query system. During training, the student model only queries the teacher when it is "uncertain" about an input. This drastically reduces training time and compute costs.
Overview of the proposed ERSAM framework.

Datasets

Because no public, realistic dataset for concurrent speaker counting existed, the authors created LibriSpeech-SAM.

  • Source: Synthesized mixtures from the LibriSpeech corpus.
  • Composition: Audio segments containing 0, 1, 2, 3, 4, or 5+ concurrent speakers.
  • Scale: Only 8 hours of training data (to simulate low-resource clinical settings) and 2 hours for testing.
  • Augmentation: They also used the TUT Acoustic Scenes dataset for non-speech samples and MUSAN/LibriAdapt to test robustness against noise and different recording devices.

Key Insights and Ideas

  • Hardware bottlenecks are specific: It’s not just about the number of parameters. As shown in the profiling of wav2vec, a single operator can dominate the latency budget if its kernel size and sequence length aren't optimized for the target mobile chip.
  • Teachers are only needed for the "hard" parts: The authors observed that larger models primarily outperform smaller ones when the smaller model is both wrong and uncertain (see Figure 5). By only querying the teacher in these specific cases, ERSAM achieves the accuracy of full KD at nearly the speed of standard training.
Fig. 4: The percentage of cases when varying the smaller model’s uncer- tainties (AC: both correct, LC: only the larger model correct, SC: only the smaller model correct, and AW: both wrong).

Key Results

ERSAM-delivered models significantly outperform previous state-of-the-art (SOTA) methods across all metrics:

  • Accuracy: Achieved a 14.3% error rate, outperforming the previous SOTA, CountNet (16.7%), and a scaled-down wav2vec (17.4%).
  • Efficiency: On a Pixel 3, ERSAM models showed a latency of only 47ms for a 5-second audio clip and an energy consumption of 38 mW·h—far below the requirements for real-time usage.
  • Training Speed: The efficient KD scheme achieved a 2.6% accuracy boost over standard training while adding only 5% to the search/training time (compared to a 195% increase for vanilla KD).
Fig. 1: DNNs searched by our ERSAM outperform state-of-the-art DNN-based SAM: achieving the best accuracy vs. hardware effi- ciency trade-offs while meeting the requirements of being energy- efficient and achieving real-time latency. Here a smaller volume of each data point’s 3D cuboid corresponds to a better accuracy vs. hardware efficiency trade-off.

Limitations and Future Work

While ERSAM shows excellent results on synthesized data and handles simulated noise well, the authors acknowledge that real-world clinical audio is more complex. Future work likely involves testing the framework on actual clinical datasets and exploring how these optimized architectures perform across a wider variety of mobile hardware beyond the Pixel 3. The current model also caps speaker counting at "5 or more," which could be expanded for even more crowded environments.

Summary updated

Abstract

Social ambiance describes the context in which social interactions happen, and can be measured using speech audio by counting the number of concurrent speakers. This measurement has enabled various mental health tracking and human-centric IoT applications. While on-device Socal Ambiance Measure (SAM) is highly desirable to ensure user privacy and thus facilitate wide adoption of the aforementioned applications, the required computational complexity of state-of-the-art deep neural networks (DNNs) powered SAM solutions stands at odds with the often constrained resources on mobile devices. Furthermore, only limited labeled data is available or practical when it comes to SAM under clinical settings due to various privacy constraints and the required human effort, further challenging the achievable accuracy of on-device SAM solutions. To this end, we propose a dedicated neural architecture search framework for Energy-efficient and Real-time SAM (ERSAM). Specifically, our ERSAM framework can automatically search for DNNs that push forward the achievable accuracy vs. hardware efficiency frontier of mobile SAM solutions. For example, ERSAM-delivered DNNs only consume 40 mW x 12 h energy and 0.05 seconds processing latency for a 5 seconds audio segment on a Pixel 3 phone, while only achieving an error rate of 14.3% on a social ambiance dataset generated by LibriSpeech. We can expect that our ERSAM framework can pave the way for ubiquitous on-device SAM solutions which are in growing demand.
BibTeX
@inproceedings{li2023ersam,
  author = {Chaojian Li and Wenwan Chen and Jiayi Yuan and Yingyan Celine Lin and Ashutosh Sabharwal},
  title = {{ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=8ySmOwbFsG},
}
Figure