TL;DR
LHGNN (Local-Higher Order Graph Neural Network) is a new architecture for audio classification that outperforms Transformers while using significantly fewer parameters. By combining local neighborhood information (via k-NN) with higher-order semantic clusters (via Fuzzy C-Means), the model captures complex audio relationships more effectively and remains highly robust even without massive ImageNet pretraining.
What Problem Does This Paper Address?
In recent years, Transformers have become the gold standard for audio tasks because their self-attention mechanisms can capture global dependencies across a spectrogram. However, the authors argue that Transformers have two major weaknesses:
- Pairwise Limitations: Self-attention focuses on interactions between pairs of points (tokens), which may fail to capture the "higher-order" relationships necessary to identify distinct, complex audio objects.
- Data Hunger: Transformers typically require extensive pretraining on massive datasets (like ImageNet) to perform well, making them less accessible for niche applications or resource-constrained environments.
The authors propose using Graph Neural Networks (GNNs) and clustering to move beyond simple pairwise interactions and create a more "data-efficient" model.
Methods
The LHGNN architecture processes a mel-spectrogram through a convolutional "stem" and then passes it through four stages of LHG Blocks. The core innovation lies in the Local-Higher Order Graph Convolution:
- Local Modeling (k-NN): For every "node" (pixel/patch in the feature map), the model identifies its $k$ nearest neighbors. This captures immediate, local context.
- Higher-Order Modeling (Fuzzy C-Means): The model uses Fuzzy C-Means clustering to group features into "centroids." Unlike standard k-means, fuzzy clustering allows a data point to belong to multiple clusters with varying degrees of membership. This helps the model recognize broader semantic categories or "audio objects."
- The Fusion Kernel: The model updates each node by concatenating the original feature with the maximum deviation from its local neighbors and the maximum deviation from its assigned cluster centroids. This "max-relative" graph convolution allows the model to see both the "trees" (local) and the "forest" (higher-order clusters).
- ConvFFN: Instead of a standard Transformer Feed-Forward Network, they use a convolutional version that includes depth-wise convolutions to preserve local spatial information across layers.
Datasets
The model was evaluated on three major benchmarks:
- AudioSet: A massive dataset of over 2 million 10-second YouTube clips across 527 classes.
- FSD50K: A dataset of over 51,000 sound event clips based on the AudioSet ontology.
- ESC50: A smaller, labeled collection of 2,000 environmental sound recordings (50 classes).
Key Insights and Ideas
- Beyond Pairwise: By integrating clustering directly into the graph convolution, the model isn't just looking at how point A relates to point B; it’s looking at how point A relates to the "concept" represented by a cluster of points.
- Inductive Bias Matters: The use of a convolutional stem and a graph-based structure provides a stronger "inductive bias" than pure Transformers. This is why the model performs so well even when it hasn't been "warmed up" on ImageNet.
- Efficiency: LHGNN achieves state-of-the-art results with 31 million parameters, compared to the 87 million parameters required by the popular Audio Spectrogram Transformer (AST).
- Robustness to "Cold Starts": On AudioSet, when training from scratch (no pretraining), LHGNN achieved a mean Average Precision (mAP) of 0.442, while the AST model plummeted to 0.366.
Key Results
- AudioSet: LHGNN achieved a top mAP of 0.466 (with pretraining), surpassing the AST model (0.459) and PSLA (0.444).
- FSD50K: It reached an mAP of 0.59, outperforming AST (0.574) and the FSD50K baseline (0.434).
- ESC50: The model achieved 96.2% accuracy, placing it among the top-performing models for environmental sound classification.
- Ablation Study: The authors proved that combining both local (k-NN) and higher-order (clusters) information was essential; using either one alone resulted in significantly lower performance (dropping mAP from 0.573 to 0.531 or 0.501).
Limitations and Future Work
- Convergence Speed: While LHGNN is parameter-efficient, it is slower to train. On AudioSet, it requires roughly 30 epochs to converge, whereas AST (with ImageNet pretraining) only needs about 5.
- Computational Complexity: Integrating clustering into every block adds computational overhead. The authors note that a more efficient method for integrating cluster centroids is needed to reduce overall computation time.
- Expansion: The authors plan to test the architecture on a broader range of audio tasks, such as music tagging and speech recognition, to further prove its versatility.
Summary updated