Biologically Motivated Audio-Visual Cue Integration for Object

Joern Anemueller, Joerg-Henrik Back, Barbara Caputo, Jie Luo, Frank W. Ohl, Francesco Orabona, Rufin Vogels, Daphna Weinshall, Alon ZweigInfoscience 2008

Abstract

Auditory and visual cues are important sensor inputs for biological and artificial systems. They provide crucial information for navigating environments, recognizing categories, animals and people. How to combine effectively these two sensory channels is still an open issue. As a step towards this goal, this paper presents a comparison between three different multi-modal integration strategies for audio-visual object category detection. We consider a high-level and a low-level cue integration approach, both biologically motivated, and we compare them with a mid-level cue integration scheme. All three integration methods are based on the least square support vector machine (LS-SVM) algorithm and state-of-the-art audio and visual feature representations. Experiments conducted on two audio-visual object categories, dogs and guitars, show that the high-level integration scheme consistently performs better than single cue methods and the other two integration schemes. These findings confirm results from neuroscience, suggesting that high-level integration is the most suitable approach for multi-modal cue integration in artificial cognitive systems.

Introduction

In both biological and artificial systems, multi-modality is a significant advantage. Combining sight and sound allows for more robust recognition in complex, real-world environments. However, the optimal architecture for this combination remains an open question in machine learning. This paper seeks to determine whether raw data concatenation (low-level), kernel-based decision functions (mid-level), or independent expert voting (high-level) is most effective for object categorization, and how these results align with neuroscientific findings.

Methods and Integration Strategies

The researchers standardized their comparison using the Least Square Support Vector Machine (LS-SVM) as the core building block. LS-SVMs provide outputs that converge toward actual class probabilities, offering a reliable "confidence score" for fusion. Three distinct strategies were compared:

1. Low-Level Integration

This approach involves simple feature concatenation. Audio and visual vectors are fused into a single high-dimensional vector before being processed by the classifier. While straightforward, this method often suffers from the "curse of dimensionality."

2. Mid-Level Integration (Multi-Cue Kernel - MCK)

This sophisticated approach uses a weighted linear combination of Mercer kernels. It allows the system to identify optimal weighting factors ($\beta$) for different cues during a single classification step.

Mid-level integration (MCK) using a weighted linear combination of kernels.

3. High-Level Integration (Discriminative Accumulation Scheme - DAS)

In the DAS approach, separate LS-SVMs are trained for audio and video. Each produces a hypothesis based on the distance from the separating hyperplane. These hypotheses are then summed using weights ($\alpha$) determined via cross-validation.

High-level integration (DAS) where separate classifiers provide confidence estimates that are later combined.

Experimental Setup

The study utilized state-of-the-art feature representations:

  • Audio: A 377-dimensional amplitude modulation spectrogram mimicking human hearing sensitivity.
  • Video: A generative relational part-based model extracting interest regions and using Discrete Cosine Transform (DCT) coefficients.

Experiments were conducted on two categories: Dogs and Guitars. To ensure a challenging task, visual data from the Caltech Dataset was paired with difficult backgrounds (e.g., roads for guitars, sites for dogs). Audio data was manually collected and mixed with environmental noise.

Key Results

The high-level DAS approach consistently outperformed all other methods across both categories. Notably, the performance gain was most significant for the "Dogs" category, which presented the most unimodal difficulty.

Method Guitars Accuracy Dogs Accuracy
Audio Only 91.11% 88.76%
Video Only 96.20% 87.64%
Low-Level 98.06% 89.91%
Mid-Level (MCK) 98.10% 91.20%
High-Level (DAS) 99.03% 93.42%

Conclusion

The success of the high-level integration scheme (DAS) suggests that multi-modal is always better than unimodal, but the method of fusion matters. The DAS approach's superiority aligns with neuroscientific evidence indicating that robust object-level categorization in the brain occurs at higher cortical levels. This confirms that high-level integration is a highly suitable approach for artificial cognitive systems aiming to mimic biological efficiency.

Summary updated

Abstract

Auditory and visual cues are important sensor inputs for biological and artificial systems. They provide crucial information for navigating environments, recognizing categories, animals and people. How to combine effectively these two sensory channels is still an open issue. As a step towards this goal, this paper presents a comparison between three different multi-modal integration strategies, for audiovisual object category detection. We consider a high-level and a low-level cue integration approach, both biologically motivated, and we compare them with a mid-level cue integration scheme. All the three integration methods are based on the least square support vector machine algorithm, and state of the art audio and visual feature representations. We conducted experiments on two audio-visual object categories, dogs and guitars, presenting different visual and auditory characteristics. Results show that the high-level integration scheme consistently performs better than single cue methods, and of the other two integration schemes. These findings confirm results from the neuroscience. This suggests that the high-level integration scheme is the most suitable approach for multi-modal cue integration for artificial cognitive systems.
BibTeX
@inproceedings{anemueller2008biologically,
  author = {Joern Anemueller and Joerg-Henrik Back and Barbara Caputo and Jie Luo and Frank W. Ohl and Francesco Orabona and Rufin Vogels and Daphna Weinshall and Alon Zweig},
  title = {{Biologically Motivated Audio-Visual Cue Integration for Object}},
  booktitle = {Infoscience},
  year = {2008},
}
Figure