Listen As You Wish: Audio based Event Detection via Text-to-Audio Grounding in Smart Cities

Haoyu Tang, Yunxiao Wang, Jihua Zhu, Shuaike Zhang, Mingzhu Xu, Qinghai Zheng, Yupeng HuarXiv 2021 · arXiv

TL;DR

The researchers introduce the Cross-modal Graph Interaction (CGI) model for text-to-audio grounding, a task that involves finding specific sound events in long, untrimmed audio files using natural language queries. By modeling word-level relationships through graphs and using a "cross-gating" mechanism to filter out noise, CGI significantly outperforms existing methods in precisely localizing complex events like traffic accidents in smart city environments.

What Problem Does This Paper Address?

In the context of smart cities and Vehicle Road Cooperation Systems (VRCS), detecting traffic events via audio is a robust alternative to cameras, which can be blinded by bad weather or poor lighting. However, traditional Sound Event Detection (SED) is limited; it can only identify sounds from a pre-defined list (e.g., "car horn").

The more advanced task, Text-to-Audio Grounding (TAG), allows users to search for complex scenarios described in plain English, such as "After a series of vehicle collisions, a sudden explosion occurs."

The challenge is that existing TAG models often treat the entire query as a single "blob" of data (a global representation). This approach fails for three reasons:

  1. It ignores the specific relationships between words (like "after" or "series of").
  2. It treats all words as equally important (even though "explosion" is more descriptive than "occurs").
  3. It doesn't effectively filter out irrelevant audio segments that don't match the query.
Figure 1: Illustration of the traffic event detection in VRCS. In this example, the left portion reveal that the vehicle-to- vehicle, vehicle-to-pedestrian, and vehicle-to-center technologies can be employed to ensure the safety control of vehicles and collaborative management of roadways after the vehicle collisions happens. The right portion presents the difference between SED and TAG in VRCS. The SED method can localize two “vehicle collision” segments in audio, yet it fails to differentiate between a singular “ vehicle collision” and “a series of vehicle collisions”. In contrast, given the query “After a series of vehicle collisions, a sudden explosion occurs.”, the TAG precisely grounds the audio segment containing the events “a series of vehicle collisions” and “explosion” with the on- and off-set (i.e., from 16.1s to 20.3s).

Methods

The authors propose the CGI model, which moves away from global representations toward a fine-grained, interactive approach. The architecture consists of four main stages:

  1. Dual Encoders: Audio is processed via a Convolutional Recurrent Neural Network (CRNN) to extract temporal features, while text is converted into word embeddings.
  2. Intra-modal Query Graph: Instead of a flat list of words, the model builds a graph where each word is a node. This allows the model to capture high-order relationships between words, regardless of how far apart they are in the sentence.
  3. Cross-modal Attention: This module calculates the relevance between specific audio snippets and specific words. It generates "snippet-specific" query representations, effectively highlighting which words are most relevant to a particular moment in the audio.
  4. Cross-gating Module: This is the "noise filter." It uses the audio features to gate (filter) the query features and vice-versa. If a word or an audio snippet doesn't find a strong match in the other modality, its influence is suppressed.
Figure 2: The structure of the proposed CGI model, which consists of the following components: two encoders that extract the audio and query features separately; an intra-modal query graph to model the implicit information between words in query; a cross-modal attention module for the fine-grained semantics; a cross-gating module which automatically emphasizes crucial parts in audio and query; a grounding module to measure the similarities between the audio snippets and query features for precisely returning the desired audio segments.

Datasets

The model was evaluated on the AudioGrounding dataset. This benchmark was constructed by merging data from AudioSet and AudioCaps. It contains:

  • 4,590 unique audio files.
  • 13,985 audio-query pairs.
  • A standard split of 12,373 pairs for training, 451 for validation, and 1,161 for testing.

Key Insights and Ideas

  • Words are not created equal: The model's attention mechanism learns that nouns and verbs (like "puppies" or "crying") carry more semantic weight for grounding than articles or auxiliary verbs.
  • Mutual Filtering: The cross-gating mechanism (shown below) is a powerful way to handle "untrimmed" audio. By letting the text "tell" the audio which parts to ignore, the model becomes much more resilient to background noise.
  • Graph-based Reasoning: By using a graph neural network for the query, the model can understand temporal logic (e.g., "A happened before B") much better than models that just average the word vectors.
Figure 3: The structure of the cross-gating module.

Key Results

The CGI model was compared against several baselines, including the original TAG model and a Match-LSTM approach. The results show a clear lead for CGI across all major metrics:

  • F1 Score: CGI achieved 33.72%, a significant improvement over the previous SOTA (Match-LSTM at 32.30%) and the baseline TAG (28.30%).
  • PSDS (Polyphonic Sound Detection Score): This threshold-independent metric saw a jump to 22.81, compared to 20.03 for the next best model.
  • Ablation Studies: The researchers proved that every piece of the puzzle matters. Removing the Query Graph or the Cross-gating module led to a measurable drop in performance, confirming that fine-grained interaction is the key to success.

Limitations and Future Work

While CGI is a step forward, the authors acknowledge several areas for improvement:

  • Architecture Evolution: They plan to explore Transformer architectures to better model long-range cross-modal dependencies.
  • Audio Modeling: Future versions might incorporate graph networks for the audio side as well, not just the text.
  • Edge Deployment: For smart city applications, latency is critical. The authors aim to design a lightweight version of the model that can run on edge devices (like smart cameras or traffic lights) rather than relying solely on the cloud.

Summary updated

Abstract

With the development of internet of things technologies, tremendous sensor audio data has been produced, which poses great challenges to audio-based event detection in smart cities. In this paper, we target a challenging audio-based event detection task, namely, text-to-audio grounding. In addition to precisely localizing all of the desired on- and off-sets in the untrimmed audio, this challenging new task requires extensive acoustic and linguistic comprehension as well as the reasoning for the crossmodal matching relations between the audio and query. The current approaches often treat the query as an entire one through a global query representation in order to address those issues. We contend that this strategy has several drawbacks. Firstly, the interactions between the query and the audio are not fully utilized. Secondly, it has not distinguished the importance of different keywords in a query. In addition, since the audio clips are of arbitrary lengths, there exist many segments which are irrelevant to the query but have not been filtered out in the approach. This further hinders the effective grounding of desired segments. Motivated by the above concerns, a novel Cross-modal Graph Interaction (CGI) model is proposed to comprehensively model the relations between the words in a query through a novel language graph. To capture the fine-grained relevances between the audio and query, a cross-modal attention module is introduced to generate snippet-specific query representations and automatically assign higher weights to keywords with more important semantics. Furthermore, we develop a cross-gating module for the audio and query to weaken irrelevant parts and emphasize the important ones.
BibTeX
@misc{tang2021listen,
  author = {Haoyu Tang and Yunxiao Wang and Jihua Zhu and Shuaike Zhang and Mingzhu Xu and Qinghai Zheng and Yupeng Hu},
  title = {{Listen As You Wish: Audio based Event Detection via Text-to-Audio Grounding in Smart Cities}},
  year = {2021},
  eprint = {2106.14136},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.14136},
}

Loading related content...

Figure