LHGNN: Local-Higher Order Graph Neural Networks For Audio Classification and Tagging

Shubhr Singh, Emmanouil Benetos, Huy Phan, Dan Stowell

TL;DR

LHGNN (Local-Higher Order Graph Neural Network) is a new architecture for audio classification that outperforms Transformers while using significantly fewer parameters. By combining local neighborhood information (via k-NN) with higher-order semantic clusters (via Fuzzy C-Means), the model captures complex audio relationships more effectively and remains highly robust even without massive ImageNet pretraining.

What Problem Does This Paper Address?

In recent years, Transformers have become the gold standard for audio tasks because their self-attention mechanisms can capture global dependencies across a spectrogram. However, the authors argue that Transformers have two major weaknesses:

  1. Pairwise Limitations: Self-attention focuses on interactions between pairs of points (tokens), which may fail to capture the "higher-order" relationships necessary to identify distinct, complex audio objects.
  2. Data Hunger: Transformers typically require extensive pretraining on massive datasets (like ImageNet) to perform well, making them less accessible for niche applications or resource-constrained environments.

The authors propose using Graph Neural Networks (GNNs) and clustering to move beyond simple pairwise interactions and create a more "data-efficient" model.

Methods

The LHGNN architecture processes a mel-spectrogram through a convolutional "stem" and then passes it through four stages of LHG Blocks. The core innovation lies in the Local-Higher Order Graph Convolution:

  • Local Modeling (k-NN): For every "node" (pixel/patch in the feature map), the model identifies its $k$ nearest neighbors. This captures immediate, local context.
  • Higher-Order Modeling (Fuzzy C-Means): The model uses Fuzzy C-Means clustering to group features into "centroids." Unlike standard k-means, fuzzy clustering allows a data point to belong to multiple clusters with varying degrees of membership. This helps the model recognize broader semantic categories or "audio objects."
  • The Fusion Kernel: The model updates each node by concatenating the original feature with the maximum deviation from its local neighbors and the maximum deviation from its assigned cluster centroids. This "max-relative" graph convolution allows the model to see both the "trees" (local) and the "forest" (higher-order clusters).
  • ConvFFN: Instead of a standard Transformer Feed-Forward Network, they use a convolutional version that includes depth-wise convolutions to preserve local spatial information across layers.

Datasets

The model was evaluated on three major benchmarks:

  • AudioSet: A massive dataset of over 2 million 10-second YouTube clips across 527 classes.
  • FSD50K: A dataset of over 51,000 sound event clips based on the AudioSet ontology.
  • ESC50: A smaller, labeled collection of 2,000 environmental sound recordings (50 classes).

Key Insights and Ideas

  • Beyond Pairwise: By integrating clustering directly into the graph convolution, the model isn't just looking at how point A relates to point B; it’s looking at how point A relates to the "concept" represented by a cluster of points.
  • Inductive Bias Matters: The use of a convolutional stem and a graph-based structure provides a stronger "inductive bias" than pure Transformers. This is why the model performs so well even when it hasn't been "warmed up" on ImageNet.
  • Efficiency: LHGNN achieves state-of-the-art results with 31 million parameters, compared to the 87 million parameters required by the popular Audio Spectrogram Transformer (AST).
  • Robustness to "Cold Starts": On AudioSet, when training from scratch (no pretraining), LHGNN achieved a mean Average Precision (mAP) of 0.442, while the AST model plummeted to 0.366.

Key Results

  • AudioSet: LHGNN achieved a top mAP of 0.466 (with pretraining), surpassing the AST model (0.459) and PSLA (0.444).
  • FSD50K: It reached an mAP of 0.59, outperforming AST (0.574) and the FSD50K baseline (0.434).
  • ESC50: The model achieved 96.2% accuracy, placing it among the top-performing models for environmental sound classification.
  • Ablation Study: The authors proved that combining both local (k-NN) and higher-order (clusters) information was essential; using either one alone resulted in significantly lower performance (dropping mAP from 0.573 to 0.531 or 0.501).

Limitations and Future Work

  • Convergence Speed: While LHGNN is parameter-efficient, it is slower to train. On AudioSet, it requires roughly 30 epochs to converge, whereas AST (with ImageNet pretraining) only needs about 5.
  • Computational Complexity: Integrating clustering into every block adds computational overhead. The authors note that a more efficient method for integrating cluster centroids is needed to reduce overall computation time.
  • Expansion: The authors plan to test the architecture on a broader range of audio tasks, such as music tagging and speech recognition, to further prove its versatility.

Summary updated

Abstract

Transformers have set new benchmarks in audio processing tasks, leveraging self-attention mechanisms to capture complex patterns and dependencies within audio data. However, their focus on pairwise interactions limits their ability to process the higher-order relations essential for identifying distinct audio objects. To address this limitation, this work introduces the Local-Higher Order Graph Neural Network (LHGNN), a graph based model that enhances feature understanding by integrating local neighbourhood information with higher-order data from Fuzzy C-Means clusters, thereby capturing a broader spectrum of audio relationships. Evaluation of the model on three publicly available audio datasets shows that it outperforms Transformer-based models across all benchmarks while operating with substantially fewer parameters. Moreover, LHGNN demonstrates a distinct advantage in scenarios lacking ImageNet pretraining, establishing its effectiveness and efficiency in environments where extensive pretraining data is unavailable.
BibTeX
@inproceedings{singh2025lhgnn,
  author = {Shubhr Singh and Emmanouil Benetos and Huy Phan and Dan Stowell},
  title = {{LHGNN: Local-Higher Order Graph Neural Networks For Audio Classification and Tagging}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=JtLknyk3lP},
}

Loading related content...

Figure