Aligning meeting recordings via adaptive fingerprinting

T. J. Tsai, Andreas StolckeInterspeech 2015

TL;DR

This paper introduces a novel "adaptive audio fingerprinting" technique that learns on-the-fly to align unsynchronized meeting recordings from sources like cell phones. By using cumulative evidence and an unsupervised learning approach, it achieves over 99% alignment accuracy with a 100ms tolerance, significantly outperforming existing methods.

What Problem Does This Paper Address?

Imagine a scenario where multiple participants in a meeting record audio on their own devices (e.g., cell phones). These recordings are likely to be unsynchronized, with varying start and end times, and potentially skewed timestamps. The goal is to automatically align these disparate recordings to create a single, high-quality summary of the entire meeting. Traditional methods like pairwise cross-correlation are computationally prohibitive for long recordings. Furthermore, relying on file timestamps is unreliable due to potential corruption, inaccuracies, or lack of precision. This paper tackles the challenge of robustly and efficiently aligning such uncoordinated audio streams.

Methods

The core of the proposed system lies in two main components: an adaptive audio fingerprint and an alignment algorithm.

  • Adaptive Audio Fingerprint Computation:
    1. Auditory Spectrogram: The audio is first converted into a log mel spectrogram.
  • Spectrotemporal Filters: Instead of using pre-defined filters (like MFCCs), this paper proposes learning "adaptive" filters. These filters are designed based on three principles:
  • Compactness: The fingerprint should represent information efficiently.
  • Robustness: The fingerprint should be resilient to noise, achieved by maximizing the variance of feature distributions.
  • Lightweight Computation: Filters should primarily use additions and subtractions.
  • Filter Design: The filter design is formulated as an optimization problem solved using eigenvalue decomposition. This process learns filters that capture temporal and spectral modulations in the audio.
  • Deltas: Delta features are computed on the spectrotemporal features to achieve invariance to volume changes, a crucial aspect when recordings are made at different distances from the speaker.
  • Thresholding and Bit Packing: The delta features are thresholded to produce binary values, which are then packed into a compact 32-bit integer fingerprint for each frame.
  • Alignment Algorithm:
    1. Initialization: The adaptive fingerprint design is learned, fingerprints are computed for all files, and a reverse index is created. One "anchor" channel is chosen as a universal time reference.
  • Best Match: The algorithm iteratively finds the best match between an unaligned recording and the currently aligned set by accumulating histograms of offset differences between matching fingerprints.
  • Fine Alignment: Once a rough match is found, a more precise alignment is performed by calculating the bit agreement rate between the candidate recording and all already aligned files.
  • Cumulative Evidence: Steps 2 and 3 are repeated, using the most recently aligned file as a query. Crucially, the algorithm retains and accumulates evidence from previous steps, allowing it to leverage stronger matches to help align weaker ones.

Datasets

The experiments were conducted on data extracted from the ICSI meeting corpus. This corpus consists of multi-channel audio recordings of 75 research group meetings, totaling approximately 72 hours. The recordings include close-talking microphones and omnidirectional tabletop microphones, offering diversity in quality and location, mimicking the scenario of participants using their own devices. The authors generated 750 alignment scenarios from this corpus by extracting random segments with controlled overlaps for evaluation.

Key Insights and Ideas

  • Adaptive Fingerprinting is Key: The paper demonstrates that learning audio fingerprints "on-the-fly" for each specific alignment scenario significantly improves performance compared to using off-the-shelf methods like the Philips fingerprint. This adaptability allows the system to better handle the unique acoustic characteristics of different meeting recordings.
  • Volume Invariance via Deltas: Using delta features on spectrotemporal features is a clever way to achieve robustness against variations in recording volume, which is a common issue when participants use their own devices at different distances.
  • Cumulative Evidence for Weak Matches: The iterative alignment process that builds upon previous matches is crucial for handling challenging scenarios where some recordings might have very little overlap or weak signal content. This approach effectively pools evidence to improve overall alignment accuracy.
  • Learned Filters Capture Rich Information: The visualization of learned filters shows they capture complex temporal and spectral modulations, highlighting the importance of context beyond just adjacent frames. The fact that many projected filter coefficients are zero also suggests an efficient and sparse representation.

Key Results

The proposed adaptive fingerprinting system achieved remarkable results:

  • High Accuracy: The system was able to achieve > 99% alignment accuracy at a 100 ms error tolerance.
  • Significant Improvement over Baseline: Compared to the well-known Philips fingerprint, the adaptive approach improved accuracy at 100 ms tolerance from 66.5% to 89.3%.
  • Benefits of Design Choices:
    • Reducing lookup bits from 32 to 16 boosted accuracy from 89.3% to 97.2%.
  • Increasing context frames from 2 to 32 further improved accuracy to 99.4%.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the learned filters are largely uncorrelated, the projection onto the {-1,0,1} lattice can introduce slight correlations. The authors accept this as a trade-off for computational efficiency. The experiments were conducted on challenging scenarios, which might be more difficult than typical use cases.
  • Future Work: Not explicitly detailed in the paper, but the success of adaptive fingerprinting and cumulative evidence suggests potential for further exploration in more complex acoustic environments or with a larger number of participants and recordings. The information-theoretic metric for fingerprint effectiveness mentioned in reference [23] could also be a direction for future research.

Summary updated

BibTeX
@inproceedings{tsai2015aligning,
  author = {T. J. Tsai and Andreas Stolcke},
  title = {{Aligning meeting recordings via adaptive fingerprinting}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure