HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, Shlomo Dubnov

Abstract

The Hierarchical Token-Semantic Audio Transformer (HTS-AT) is an efficient architecture designed to overcome the computational and localization limitations of previous audio transformers. By integrating a hierarchical structure with patch-merging and a novel token-semantic module, HTS-AT achieves state-of-the-art performance on major benchmarks like AudioSet (0.471 mAP) and ESC-50 (97.0% accuracy). Notably, it achieves these results with only 35% of the parameters and 15% of the training time required by the Audio Spectrogram Transformer (AST), while significantly improving sound event localization capabilities.

Introduction

The Audio Spectrogram Transformer (AST) recently replaced Convolutional Neural Networks (CNNs) as the standard for sound classification. However, AST faces three primary challenges: high computational cost due to global attention on long sequences, heavy dependency on ImageNet pretraining, and poor temporal localization (the "where" problem) due to its reliance on a single class token. HTS-AT addresses these issues by introducing a scalable, efficient transformer architecture capable of precise sound event detection.

Methods

HTS-AT incorporates three main architectural innovations inspired by the Swin Transformer:

  • Hierarchical Structure with Patch-Merging: Instead of fixed sequence lengths, HTS-AT uses "Patch-Merge" layers to downsample sequence length while increasing feature dimensions as data moves deeper into the network. This reduces GPU memory consumption exponentially.
  • Window Attention: To avoid the $O(N^2)$ complexity of global attention, HTS-AT calculates attention within local windows of time and frequency, drastically speeding up computation.
  • Token-Semantic Module: A specialized CNN-based module replaces the standard class token. It maps final output tokens into an "Event Presence Map," enabling the model to predict both the identity and the temporal boundaries of sound events.
HTS-AT Architecture: Illustrating the hierarchical structure, patch-merging, and the token-semantic module for event localization.

Results and Discussion

The model was evaluated on AudioSet, ESC-50, Speech Command V2, and DESED. HTS-AT set new state-of-the-art records across multiple benchmarks while maintaining a significantly smaller footprint.

Performance comparison on AudioSet: HTS-AT variants achieve superior mAP with significantly fewer parameters compared to AST.

Key findings include:

  • AudioSet Performance: HTS-AT achieved a SOTA mAP of 0.471, outperforming larger models like AST.
  • Efficiency and Scalability: With only 31M parameters (compared to AST's 87M), HTS-AT reduces training time from 600 hours to 80 hours. This efficiency allows for larger batch sizes and faster iteration cycles.
  • Reduced Pretraining Bias: Unlike AST, HTS-AT is less dependent on ImageNet pretraining, showing only a 1-2% performance drop when trained from scratch, suggesting it learns intrinsic audio patterns more effectively.
  • Localization Accuracy: On the DESED dataset, HTS-AT outperformed specialized CNN models in event localization F1-scores, demonstrating the effectiveness of the token-semantic module.

Conclusion

HTS-AT demonstrates that audio transformers can be made leaner and more capable by leveraging hierarchical processing and semantic token mapping. While it shows some limitations in specific noisy environments (e.g., "Speech" or "Cleaner" classes in DESED), its efficiency and localization accuracy mark a significant advancement. Future work will explore training with temporally-strong labels and applying the architecture to tasks like source separation or music generation.

Summary updated

Abstract

Audio classification is an important task of mapping audio samples into their corresponding labels. Recently, the transformer model with self-attention mechanisms has been adopted in this field. However, existing audio transformers require large GPU memories and long training time, meanwhile relying on pretrained vision models to achieve high performance, which limits the model’s scalability in audio tasks. To combat these problems, we introduce HTS-AT: an audio transformer with a hierarchical structure to reduce the model size and training time. It is further combined with a token-semantic module to map final outputs into class featuremaps, thus enabling the model for the audio event detection (i.e. localization in time). We evaluate HTS-AT on three datasets of audio classification where it achieves new state-of-the-art (SOTA) results on AudioSet and ESC50, and equals the SOTA on Speech Command V2. It also achieves better performance in event localization than the previous CNN-based models. Moreover, HTS-AT requires only 35% model parameters and 15% training time of the previous audio transformer. These results demonstrate the high performance and high efficiency of HTS-AT.
BibTeX
@inproceedings{chen2022hts,
  author = {Ke Chen and Xingjian Du and Bilei Zhu and Zejun Ma and Taylor Berg-Kirkpatrick and Shlomo Dubnov},
  title = {{HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=Pl9P6WLUD8},
}
Figure