Audio-visual tracking by density approximation in a sequential Bayesian filtering framework

Israel D. Gebru, Christine Evers, Patrick A. Naylor, Radu Horaud

TL;DR

This paper introduces a novel audio-visual tracking system that fuses sound direction-of-arrival (DOA) and visual face detection to estimate the trajectories of multiple people. By representing the complex posterior probability distribution as a Gaussian Mixture Model (GMM) and employing techniques like the Unscented Transform and Expectation-Maximization, the system achieves more robust and accurate tracking, especially in challenging scenarios with occlusions.

What Problem Does This Paper Address?

Accurately tracking multiple people in real-world environments is a significant challenge, particularly for human-robot interaction where understanding who is speaking and where they are is crucial. Traditional visual trackers struggle with occlusions, poor lighting, and dynamic movements. Acoustic tracking, while good at estimating sound direction, can suffer from reverberation and track deletions during speech inactivity. This paper tackles the problem of fusing these two modalities to overcome their individual limitations and achieve more reliable multi-person tracking. The core challenge lies in the inherent non-linearity and multi-modality of audio-visual data, making traditional Bayesian filtering methods intractable.

Methods

The authors propose a sequential Bayesian filtering framework to model the audio-visual tracking problem. The key innovations lie in how they represent and update the probability distributions:

  • Gaussian Mixture Model (GMM) for Posterior Representation: Instead of a single Gaussian, the posterior probability distribution (representing the likelihood of target locations) is modeled as a GMM. This allows for representing more complex, multi-modal distributions.
  • Unscented Transform (UT) for Predictive Density: To maintain the GMM representation over time, the predictive density (how the state evolves) is approximated by a GMM using the Unscented Transform. The UT is effective at propagating the mean and covariance of a distribution through non-linear functions, which is essential for state transitions.
  • Density Interpolation for Observation Likelihood: A density interpolation technique is used to create a continuous representation of the observation likelihood (how well current observations match the predicted state), also represented as a GMM.
  • Expectation-Maximization (EM) for GMM Compression: To prevent the number of GMM components from growing exponentially with each time step, an EM algorithm is applied. This clusters similar Gaussians, resulting in a more compact and computationally manageable GMM representation of the posterior.
  • Geometric Transformation: Auditory DOA estimates are transformed into the same pixel space as visual detections, enabling a unified representation for fusion.

Datasets

The authors evaluated their approach using recordings from a camcorder and a microphone array in a controlled room setup. Two specific scenarios (S1 and S2) were used, involving two talkers speaking and moving within the room. The ground truth was established by hand-annotating bounding boxes around faces and inferring 2D locations. No new datasets were introduced; existing recordings were used for evaluation.

Key Insights and Ideas

  • Constructive Fusion is Key: The paper strongly emphasizes that simply using audio and visual data independently isn't enough. True benefit comes from fusing them to disambiguate situations where one modality is weak (e.g., visual occlusions) by leveraging the strengths of the other.
  • GMMs are Essential for Complexity: The inherent multi-modal nature of audio-visual tracking necessitates a flexible representation like GMMs. This allows the model to represent multiple potential locations or states simultaneously.
  • Managing GMM Growth is Crucial: The exponential growth of GMM components is a major computational hurdle. The EM-based compression is a critical step for making the approach practical for real-time or near-real-time applications.
  • Robustness to Missing Data: The likelihood function is designed to be robust to missing detections, meaning it doesn't penalize the model heavily if audio or visual data is temporarily unavailable, which is common in real-world scenarios.

Key Results

The proposed audio-visual tracking approach (P-AV) demonstrated significant improvements over purely visual trackers and a baseline audio-visual tracker (P-V).

  • Scenario S1: P-AV achieved a 11.3 percentage point improvement in MOTA (Multiple Object Tracking Accuracy) compared to P-V. It outperformed the benchmark visual tracker (OV) by a substantial 37.9 percentage points in MOTA and 32.7 percentage points in MOTP (Multiple Object Tracking Precision).
  • Scenario S2: P-AV showed a 3.5 percentage point improvement in MOTA over P-V and outperformed OV by 45.5 percentage points in MOTA and 32.7 percentage points in MOTP.
  • The paper attributes these gains to the ability of the GMM-based approach to handle ambiguities and occlusions more effectively than single-hypothesis trackers.

Limitations and Future Work

  • Computational Cost: While the EM algorithm helps manage GMM growth, the overall computational complexity can still be a concern, especially with a large number of targets or complex scenes.
  • Initialization: The paper mentions that face detection is used for initialization, and the approach is robust to missing detections. However, the initial performance and robustness to challenging initializations (e.g., multiple people close together) are not deeply explored.
  • Future Work: The authors suggest extending the framework to more complex scenarios, potentially involving more people or more dynamic environments. Further research into more efficient GMM compression techniques or alternative density approximations could also be beneficial.

Summary updated

Abstract

This paper proposes a novel audio-visual tracking approach that exploits constructively audio and visual modalities in order to estimate trajectories of multiple people in a joint state space. The tracking problem is modeled using a sequential Bayesian filtering framework. Within this framework, we propose to represent the posterior density with a Gaussian Mixture Model (GMM). To ensure that a GMM representation can be retained sequentially over time, the predictive density is approximated by a GMM using the Unscented Transform. While a density interpolation technique is introduced to obtain a continuous representation of the observation likelihood, which is also a GMM. Furthermore, to prevent the number of mixtures from growing exponentially over time, a density approximation based on the Expectation Maximization (EM) algorithm is applied, resulting in a compact GMM representation of the posterior density. Recordings using a camcorder and microphone array are used to evaluate the proposed approach, demonstrating significant improvements in tracking performance of the proposed audio-visual approach compared to two benchmark visual trackers.
BibTeX
@inproceedings{gebru2017audio,
  author = {Israel D. Gebru and Christine Evers and Patrick A. Naylor and Radu Horaud},
  title = {{Audio-visual tracking by density approximation in a sequential Bayesian filtering framework}},
  booktitle = {HSCMA},
  year = {2017},
  url = {https://openreview.net/forum?id=pOkefJT7cn},
}
Figure