TL;DR
This paper introduces a novel platform for transcribing meetings using participants' own mobile phones, eliminating the need for specialized infrastructure. They explore and compare three methods for combining audio from multiple devices to improve speech recognition accuracy, finding that early microphone selection based on signal quality offers a good balance of performance and computational efficiency.
What Problem Does This Paper Address?
Meetings are a crucial part of professional life, yet the information exchanged is often poorly preserved, relying on time-consuming manual note-taking that misses much of the detail. Existing solutions for automatic meeting transcription typically require dedicated infrastructure, limiting their flexibility. This paper tackles the challenge of capturing meeting content reliably and unobtrusively by leveraging the ubiquitous presence of mobile phones. The core problem is improving speech recognition accuracy in a challenging "far-field" environment (speech captured at a distance) using distributed, unsynchronized audio sources from multiple mobile devices, without requiring additional hardware.
Methods
The authors propose a three-step platform for meeting transcription using clusters of mobile devices:
- Signal Quality Estimation: Assessing the quality of audio captured by each device.
- Device Selection (Negotiation): Choosing the best device(s) for transcription.
- Combination: Merging information from selected devices to improve recognition.
They investigate three specific approaches for the selection and combination steps:
- SNR-based Microphone Selection: Selecting the single device with the highest Signal-to-Noise Ratio (SNR) for each speech segment. This is a straightforward approach that leverages a common audio quality metric.
- Class Separability Measure (CSM)-based Microphone Selection: Using a more sophisticated metric based on acoustic class separability (how well different speech sounds can be distinguished) derived from MFCC features. This aims to find channels that provide more discriminative features for recognition.
- Dynamic Confusion Network Combination (CNC): Performing speech recognition on audio from all devices, generating word hypotheses (potential transcriptions), and then combining these hypotheses using confusion networks. This approach dynamically weights the contribution of each channel based on its SNR, aiming to leverage the strengths of multiple devices.
Datasets
- AMI Meeting Corpus: Used for simulated experiments, where four far-field channels from each session were selected to mimic the setup.
- Mobile Device Meeting Corpus: A newly collected corpus for evaluating performance on actual mobile devices. This corpus consists of speech recorded in a quiet meeting room using high-end cellphones, with speakers reading news stories.
Key Insights and Ideas
- Leveraging Ubiquitous Devices: The fundamental idea that participants' own mobile phones can form a powerful, distributed transcription system is a key contribution, offering flexibility and reducing infrastructure costs.
- Early Selection is Efficient: Selecting the best microphone(s) before full transcription (at the signal or feature level) is significantly more computationally efficient and bandwidth-friendly than combining full transcription outputs from all devices.
- SNR is a Strong Baseline: Simple SNR-based microphone selection provides a substantial improvement over using a single, average far-field microphone and is surprisingly effective, even outperforming more complex methods in some scenarios.
- Trade-offs in Combination: While combining outputs (CNC) can yield the highest accuracy, it comes at a significant computational and bandwidth cost. Dynamic weighting in CNC, based on SNR, improves its effectiveness.
Key Results
- AMI Corpus (Simulated):
- SNR-based selection improved accuracy by 12% compared to the average far-field case.
- CSM-based selection achieved a 4% relative improvement over the average far-field case.
- Dynamic CNC achieved 29.6% accuracy, outperforming SNR and CSM, and was significantly better than traditional CNC.
- Mobile Device Corpus (Real-world):
- SNR-based selection achieved a 16% relative improvement in ASR accuracy compared to the average far-field microphone.
- SNR-based selection also outperformed CSM by 3.6%.
- Dynamic CNC achieved slightly better accuracy than SNR-based selection (2% relative improvement).
- Interestingly, SNR-based selection performed better than manually selecting the closest microphone, suggesting that device orientation and placement relative to the speaker are not always the best indicators of signal quality.
Limitations and Future Work
- Limitations: The paper acknowledges that the CSM approach's effectiveness was somewhat limited by the amount of training data and the difficulty in reliably selecting silence classes. The real-world evaluation used a controlled setup with high-end cellphones, which might not fully represent the diversity of devices and environments encountered in spontaneous meetings.
- Future Work: The authors aim to develop a real-time implementation of their proposed approach. Key challenges for future work include achieving accurate synchronization across devices, robust speech/non-speech detection, and effectively handling multiple overlapping speakers (barge-in effects).
Summary updated