Abstract
The Hierarchical Token-Semantic Audio Transformer (HTS-AT) is an efficient architecture designed to overcome the computational and localization limitations of previous audio transformers. By integrating a hierarchical structure with patch-merging and a novel token-semantic module, HTS-AT achieves state-of-the-art performance on major benchmarks like AudioSet (0.471 mAP) and ESC-50 (97.0% accuracy). Notably, it achieves these results with only 35% of the parameters and 15% of the training time required by the Audio Spectrogram Transformer (AST), while significantly improving sound event localization capabilities.
Introduction
The Audio Spectrogram Transformer (AST) recently replaced Convolutional Neural Networks (CNNs) as the standard for sound classification. However, AST faces three primary challenges: high computational cost due to global attention on long sequences, heavy dependency on ImageNet pretraining, and poor temporal localization (the "where" problem) due to its reliance on a single class token. HTS-AT addresses these issues by introducing a scalable, efficient transformer architecture capable of precise sound event detection.
Methods
HTS-AT incorporates three main architectural innovations inspired by the Swin Transformer:
- Hierarchical Structure with Patch-Merging: Instead of fixed sequence lengths, HTS-AT uses "Patch-Merge" layers to downsample sequence length while increasing feature dimensions as data moves deeper into the network. This reduces GPU memory consumption exponentially.
- Window Attention: To avoid the $O(N^2)$ complexity of global attention, HTS-AT calculates attention within local windows of time and frequency, drastically speeding up computation.
- Token-Semantic Module: A specialized CNN-based module replaces the standard class token. It maps final output tokens into an "Event Presence Map," enabling the model to predict both the identity and the temporal boundaries of sound events.
Results and Discussion
The model was evaluated on AudioSet, ESC-50, Speech Command V2, and DESED. HTS-AT set new state-of-the-art records across multiple benchmarks while maintaining a significantly smaller footprint.
Key findings include:
- AudioSet Performance: HTS-AT achieved a SOTA mAP of 0.471, outperforming larger models like AST.
- Efficiency and Scalability: With only 31M parameters (compared to AST's 87M), HTS-AT reduces training time from 600 hours to 80 hours. This efficiency allows for larger batch sizes and faster iteration cycles.
- Reduced Pretraining Bias: Unlike AST, HTS-AT is less dependent on ImageNet pretraining, showing only a 1-2% performance drop when trained from scratch, suggesting it learns intrinsic audio patterns more effectively.
- Localization Accuracy: On the DESED dataset, HTS-AT outperformed specialized CNN models in event localization F1-scores, demonstrating the effectiveness of the token-semantic module.
Conclusion
HTS-AT demonstrates that audio transformers can be made leaner and more capable by leveraging hierarchical processing and semantic token mapping. While it shows some limitations in specific noisy environments (e.g., "Speech" or "Cleaner" classes in DESED), its efficiency and localization accuracy mark a significant advancement. Future work will explore training with temporally-strong labels and applying the architecture to tasks like source separation or music generation.
Summary updated