Audio-Visual Calibration with Polynomial Regression for 2-D Projection Using SVD-PHAT

François Grondin, Hao Tang, James R. GlassICASSP 2020 · DOI

Abstract

This paper introduces a streamlined method to overlay real-time acoustic "heat maps" onto standard video using low-cost hardware. By using polynomial regression to map sound arrival times directly to image pixels and leveraging a computationally efficient algorithm called SVD-PHAT, the researchers achieved a 47x speedup in processing, enabling smooth audio-visual alignment on standard consumer laptops.

Introduction

In applications like robotics or automated video conferencing, it is incredibly useful to "see" where a sound is coming from. While algorithms for sound source localization (SSL) exist, aligning that auditory data with a camera's visual field is notoriously difficult. Traditional calibration often requires expensive "acoustic cameras," specialized hardware, or complex 3D modeling to account for lens distortion.

Furthermore, the standard method for generating acoustic images—Steered-Response Power Phase Transform (SRP-PHAT)—is computationally "heavy," making it difficult to run in real-time on embedded systems. The authors sought a way to calibrate a simple microphone array and camera setup quickly and without specialized equipment.

Methods: Direct 2D Mapping and SVD-PHAT

The authors propose a two-pronged approach to simplify the audio-visual alignment process:

  1. Direct Polynomial Regression: Instead of mapping sound to a 3D coordinate and then projecting that 3D point onto a 2D image, they map the Time Difference of Arrival (TDOA) of sound between microphone pairs directly to the $(u, v)$ pixel coordinates of the image. This direct mapping implicitly handles non-linearities, such as "barrel" or "pincushion" lens distortion, without requiring knowledge of the camera's internal parameters.
Conceptual diagram of the audio-visual calibration setup, showing the mapping of target points to image pixels.
  1. SVD-PHAT for Efficiency: To generate the acoustic heat map, they use Singular Value Decomposition with Phase Transform (SVD-PHAT). This technique decomposes the massive projection matrix used in sound localization into smaller, manageable parts, significantly reducing the number of complex multiplications required to calculate sound energy for every pixel.

The Calibration Process: The setup is remarkably "low-tech." A user simply stands in front of the camera and makes a consistent sound (like a "hissing" /s/ sound) at specific target points shown on a screen. The system records the TDOAs at these points and calculates the regression coefficients.

Key Insights and Results

Through simulation and experimental testing with a ReSpeaker 4-microphone array and an Arducam 5MP camera, the authors identified several key advantages of their approach:

  • The "Sweet Spot" for Complexity: Simulations showed that while linear and quadratic regressions fail to model TDOAs accurately under lens distortion, a third-order polynomial ($L=3$) provides a dramatic increase in accuracy, effectively neutralizing common optical distortions.
  • Massive Computational Savings: The transition from standard SRP-PHAT to SVD-PHAT achieved a 47x reduction in computational complexity. This turned a task requiring over 118 million complex multiplications per frame into one requiring only about 2.5 million.
  • Real-time Performance: The system successfully generated acoustic images at a QVGA resolution ($320 \times 240$) in real-time on a standard laptop with an Intel Core i5 processor.
Qualitative results showing (a) the calibration phase and the resulting heat maps for (b) speech, (c) a smartphone playing music, and (d) finger snapping.

Qualitative results confirmed that the heat map accurately centers on various sound sources, including speech, music from a smartphone, and finger snapping. Quantitative simulations confirmed that the $L=3$ regression maintained a low Root Mean Square Error (RMSE) even with significant lens distortion.

Limitations and Future Work

The authors note that the "center of mass" of the acoustic heat map is occasionally slightly shifted, likely due to the small aperture of the low-cost microphone array, which is sensitive to tiny phase shifts and noise. Future work will involve integrating this fast calibration and imaging method into more complex tasks, such as multimodal segmentation (isolating a speaker in a crowded video) and improved audio-visual classification for robotics.

Summary updated

BibTeX
@inproceedings{grondin2020audio,
  author = {François Grondin and Hao Tang and James R. Glass},
  title = {{Audio-Visual Calibration with Polynomial Regression for 2-D Projection Using SVD-PHAT}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9054690},
}
Figure