Abstract
This paper proposes PickNet, a neural network model for real-time channel selection for an ad hoc microphone array consisting of multiple recording devices like cell phones. Assuming at most one person to be vocally active at each time point, PickNet identifies the device that is spatially closest to the active person for each time frame by using a short spectral patch of just hundreds of milliseconds. The model is applied to every time frame, and the short time frame signals from the selected microphones are concatenated across the frames to produce an output signal. As the personal devices are usually held close to their owners, the output signal is expected to have higher signal-to-noise and direct-to-reverberation ratios on average than the input signals. Since PickNet utilizes only limited acoustic context at each time frame, the system using the proposed model works in real time and is robust to changes in acoustic conditions. Speech recognition-based evaluation was carried out by using real conversational recordings obtained with various smartphones. The proposed model yielded significant gains in word error rate with limited computational cost over systems using a block-online beamformer and a single distant microphone.
Introduction
In modern workplace environments, meetings often involve multiple personal devices such as smartphones, laptops, and tablets. While these devices collectively form an "ad hoc microphone array," leveraging them effectively for speech recognition is challenging. The number of devices is unpredictable, their spatial geometry is unknown, and they often possess varying hardware characteristics.
Existing techniques, such as blind beamforming or batch-processed channel selection, often require "looking ahead" at the entire recording or involve heavy computations that make them unsuitable for live applications. Simple heuristics, like selecting the channel with the highest energy, frequently fail in noisy or reverberant rooms. PickNet addresses these issues by providing a real-time, flexible, and robust channel selection mechanism.
The PickNet Architecture
The core of the proposed system is PickNet, a convolutional neural network (CNN) designed to estimate the probability of each microphone being the closest to the current speaker.
Cross-Channel Layers
A key innovation in PickNet is the use of cross-channel layers. While standard CNNs process each channel independently, PickNet employs a "transform-aggregate-concatenate" approach. It extracts local features from each channel, averages them across all channels to create a "global" context, and then concatenates this global information back into each local channel. This allows the model to compare microphones relative to one another rather than in isolation.
Permutation Invariance
By utilizing a mean pooling step for aggregation, the model maintains permutation invariance. This ensures that the system produces consistent results regardless of the order in which devices are registered, which is essential for ad hoc setups where device order is arbitrary.
Efficiency via Frame Subsampling
To minimize computational overhead, the model implements frame subsampling, running every $N=3$ frames (approximately every 48ms). Given that human speech patterns and speaker locations do not change instantaneously, this reduction in evaluation frequency yields a 66% decrease in computational cost with negligible impact on Word Error Rate (WER).
Experimental Evaluation
The model was trained using clean speech from the WSJ1 and LibriSpeech corpora, augmented with simulated "shoebox" room acoustics and noise from the MUSAN corpus. Evaluation was performed on 12 real conversational recordings involving 2–4 speakers using various smartphones in diverse settings, ranging from quiet conference rooms to a noisy atrium.
Key Results
- Accuracy: PickNet significantly improved performance, reducing the WER from 16.4% (using a single distant microphone) to 12.6% (hand-held) and 13.8% (on-table).
- Competitive Performance: Despite its real-time constraints, PickNet performed on par with or better than non-real-time, batch-processed methods like ROVER or M-measure.
- Computational Speed: On a standard laptop CPU, PickNet processes one frame of audio in just 1.32 ms (for a 3-mic setup), making it well-suited for mobile or edge device deployment.
Conclusion
PickNet demonstrates that a lightweight neural network can effectively perform real-time channel selection in complex, ad hoc environments. By focusing on spatial cues through cross-channel comparisons, it achieves high accuracy with minimal latency. Future research will address the challenges of overlapped speech and the integration of device diarization into the workflow.
Summary updated