Abstract
Transformers have set new benchmarks in audio processing tasks, leveraging self-attention mechanisms to capture complex patterns and dependencies within audio data. However, their focus on pairwise interactions limits their ability to process the higher-order relations essential for identifying distinct audio objects. To address this limitation, this work introduces the Local-Higher Order Graph Neural Network (LHGNN), a graph-based model that enhances feature understanding by integrating local neighborhood information with higher-order data from Fuzzy C-Means clusters, thereby capturing a broader spectrum of audio relationships. Evaluation of the model on three publicly available audio datasets shows that it outperforms Transformer-based models across all benchmarks while operating with substantially fewer parameters. Moreover, LHGNN demonstrates a distinct advantage in scenarios lacking ImageNet pretraining, establishing its effectiveness and efficiency in environments where extensive pretraining data is unavailable.
Introduction
In recent years, the Audio Spectrogram Transformer (AST) and similar architectures have become the standard for audio classification. By treating spectrograms as sequences of patches and applying self-attention, these models capture global dependencies effectively. However, Transformers face two significant hurdles: they are inherently limited by pairwise token interactions, which may overlook complex, higher-order semantic structures, and they are notoriously "data-hungry," often requiring extensive pretraining on ImageNet to achieve competitive performance.
To overcome these challenges, the Local-Higher Order Graph Neural Network (LHGNN) shifts the paradigm from global self-attention to a graph-based approach. By combining local neighborhood context with higher-order semantic clustering, LHGNN provides a more nuanced understanding of audio objects while maintaining high parameter efficiency.
The LHGNN Architecture
The LHGNN architecture begins with a convolutional "stem" that processes the input mel-spectrogram. The resulting feature maps are then passed through four stages of LHG Blocks. The innovation of the model lies in its unique Graph Convolutional layer, which operates on two distinct scales:
1. Local Modeling via k-NN
For every node (representing a pixel or patch in the feature map), the model identifies its $k$ nearest neighbors. This ensures that the model captures immediate spatial and temporal context, allowing it to resolve fine-grained details within the audio signal.
2. Higher-Order Modeling via Fuzzy C-Means
To capture broader semantic relationships, LHGNN employs Fuzzy C-Means (FCM) clustering. Unlike traditional hard clustering, FCM allows data points to belong to multiple clusters with varying degrees of membership. These clusters represent "audio objects" or abstract semantic concepts. By relating individual nodes to these cluster centroids, the model gains a "higher-order" perspective that transcends simple pairwise comparisons.
3. The Fusion Kernel and ConvFFN
The model updates node features using a "max-relative" graph convolution. It concatenates the original feature with the maximum deviation from its local neighbors and the maximum deviation from its assigned cluster centroids. This allows the network to simultaneously observe the "trees" (local context) and the "forest" (semantic clusters). Finally, a Convolutional Feed-Forward Network (ConvFFN) replaces the standard Transformer FFN, utilizing depth-wise convolutions to preserve spatial information across layers.
Experimental Evaluation
The researchers evaluated LHGNN against state-of-the-art models on three major benchmarks: AudioSet, FSD50K, and ESC50.
Performance Benchmarks
- AudioSet: LHGNN achieved a mean Average Precision (mAP) of 0.466, surpassing the Audio Spectrogram Transformer (AST) at 0.459 and PSLA at 0.444.
- FSD50K: The model reached an mAP of 0.59, a significant improvement over the AST (0.574) and the FSD50K baseline (0.434).
- ESC50: For environmental sound classification, the model achieved 96.2% accuracy, placing it among the top-tier architectures for this task.
Efficiency and Robustness
A key highlight of LHGNN is its efficiency. The model achieves these results with only 31 million parameters, a stark contrast to the 87 million parameters required by AST.
Furthermore, LHGNN demonstrates remarkable robustness in "cold start" scenarios. When trained on AudioSet from scratch without any ImageNet pretraining, LHGNN maintained a high mAP of 0.442. In the same conditions, the AST model's performance dropped significantly to 0.366, highlighting LHGNN's superior inductive bias and data efficiency.
Discussion and Ablation
Ablation studies confirmed that the integration of both local and higher-order information is critical. Removing the clustering component or the k-NN component resulted in a performance drop (e.g., on FSD50K, mAP fell from 0.573 to 0.531 or 0.501 when using only one of the two). This proves that the synergy between local neighborhood data and semantic clusters is the primary driver of the model's success.
However, the model is not without trade-offs. The integration of clustering into every block increases computational complexity, leading to slower training times. While AST (with pretraining) can converge in roughly 5 epochs on AudioSet, LHGNN requires approximately 30 epochs to reach its peak performance.
Conclusion
LHGNN represents a significant step forward in audio classification by moving beyond the pairwise limitations of self-attention. By leveraging graph neural networks and fuzzy clustering, it captures complex audio relationships more effectively than Transformers while remaining parameter-efficient and robust to the absence of large-scale pretraining. Future work will focus on optimizing the computational overhead of the clustering mechanism and extending the architecture to other domains such as music tagging and speech recognition.
Summary updated