Picknet: Real-Time Channel Selection for Ad Hoc Microphone Arrays

Takuya Yoshioka, Xiaofei Wang, Dongmei Wang

TL;DR

PickNet is a lightweight neural network designed to select the "best" microphone in real-time from an ad hoc array of devices, such as smartphones scattered across a meeting table. By identifying the device spatially closest to the active speaker every few milliseconds, it significantly improves speech recognition accuracy with minimal computational overhead and a tiny 64ms latency.

What Problem Does This Paper Address?

In modern workplaces, meetings often involve multiple personal devices (phones, laptops, tablets) equipped with microphones. While these devices form an "ad hoc microphone array," leveraging them effectively is difficult. The number of devices is unpredictable, their geometry is unknown, and they may have different hardware characteristics.

Existing techniques like blind beamforming or batch-processed channel selection often require "looking ahead" at the entire recording or involve heavy computations that make them unsuitable for live applications. Simple methods, like picking the channel with the highest energy, often fail in noisy or reverberant rooms. The authors aim to create a system that is:

  1. Real-time: Low latency and low CPU usage.
  2. Flexible: Capable of handling a varying number of microphones in any order.
  3. Robust: Effective in real-world acoustic environments with noise and reverberation.

Methods

The core of the paper is PickNet, a convolutional neural network (CNN) that estimates the probability of each microphone being the closest to the current speaker.

  • Cross-Channel Layers: This is the primary innovation. Standard CNNs process each channel independently. PickNet uses a "transform-aggregate-concatenate" approach. It extracts local features from each channel, averages them across all channels to create a "global" context, and then concatenates this global info back into each local channel. This allows the model to compare microphones rather than just looking at them in isolation.
  • Permutation Invariance: Because of the averaging (mean pooling) step, the model produces the same result regardless of the order in which the microphones are plugged into the system—a crucial requirement for ad hoc setups.
  • Frame Subsampling: To save power and CPU, the model only runs every $N=3$ frames (approx. every 48ms). Since humans don't move or switch turns instantly, the "best" microphone doesn't change fast enough to require frame-by-frame evaluation.
  • Loss Function: The model is trained using Mean Squared Error (MSE) between the selected "enhanced" signal and the cleanest possible near-field signal, effectively teaching the network to minimize distortion and noise.

Datasets

  • Training: The authors used WSJ1 and LibriSpeech as clean speech sources. They simulated thousands of virtual "shoebox" rooms with varying dimensions, reverberation times ($T_{60}$), and noise levels (using the MUSAN corpus) to create a robust synthetic training set.
  • Evaluation: Unlike many papers that rely solely on simulations, this study evaluated PickNet on 12 real conversational recordings. These involved 2–4 speakers using various iOS and Android smartphones in different settings, from small conference rooms to a noisy atrium.

Key Insights and Ideas

  • Comparison is Better than Isolation: The "Model without cross-channel layers" performed worse. This confirms that a neural network needs to see the "big picture" of all available audio streams to decide which one is truly the cleanest.
  • Spatial over Spectral: By using a short spectral patch (hundreds of milliseconds), the model focuses on spatial cues (which mic is closer) rather than trying to perform complex speech enhancement, which is harder to do in real-time.
  • Efficiency via Subsampling: Reducing the evaluation frequency by a factor of three resulted in a 66% reduction in computational cost with almost zero impact on Word Error Rate (WER).

Key Results

  • Accuracy: PickNet reduced the Word Error Rate (WER) from 16.4% (using a single distant microphone) to 12.6% (hand-held) and 13.8% (on-table).
  • Competitive Performance: PickNet performed on par with or better than non-real-time, batch-processed methods like ROVER or M-measure, despite having much lower latency.
  • Speed: On a standard laptop CPU, PickNet takes only 1.32 ms to process one frame of audio (in a 3-mic setup), making it well-suited for mobile or edge device deployment.

Limitations and Future Work

  • Overlapped Speech: The current model assumes only one person is talking at a time. If two people speak simultaneously, the model may struggle or jump between channels. The authors explicitly list handling overlapped speech as a goal for future work.
  • Device Diarization: While the system identifies the best microphone, it still relies on a separate "device diarization" step to assign speaker labels to the recognized words.
  • Hardware Variance: While tested with different smartphones, the impact of vastly different microphone qualities (e.g., a high-end laptop vs. a budget phone) wasn't the primary focus of this study.

Summary updated

Abstract

This paper proposes PickNet, a neural network model for real-time channel selection using an ad hoc microphone array. Assuming at most one person to be vocally active at each time point, PickNet identifies the device that is spatially closest to the active person for each time frame by using a short spectral patch of just hundreds of milliseconds. The model is applied to every time frame, and the short time frame signals from the selected microphones are concatenated across the frames to produce an output signal. As the personal devices are usually held close to their owners, the output signal is expected to have higher signal-to-noise and direct-to-reverberation ratios on average than the input signals. Since PickNet utilizes only limited acoustic context at each time frame, the system using the proposed model works in real time and is robust to changes in acoustic conditions. Speech recognition-based evaluation was carried out by using real conversational recordings obtained with various smart-phones. The proposed model yielded significant gains in word error rate with limited computational cost over systems using a block-online beamformer and a single distant microphone.
BibTeX
@inproceedings{yoshioka2022picknet,
  author = {Takuya Yoshioka and Xiaofei Wang and Dongmei Wang},
  title = {{Picknet: Real-Time Channel Selection for Ad Hoc Microphone Arrays}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=F-Suxena95},
}

Loading related content...

Figure