BattleSound: A Game Sound Benchmark for the Sound-Specific Feedback Generation in a Battle Game

Sungho Shin, Seongju Lee, Changhyun Jun, Kyoobin LeeSensors 2023 · DOI

TL;DR

This paper introduces BattleSound, a novel, high-resolution dataset of game sounds from PlayerUnknown's Battlegrounds (PUBG), designed to improve real-time sound event detection for haptic feedback in noisy gaming environments. They demonstrate that deep learning models trained on this dataset can accurately identify weapon sounds and voice chat, outperforming traditional methods and highlighting the critical role of short annotation intervals for effective real-time inference.

What Problem Does This Paper Address?

Immersive gaming experiences are increasingly enhanced by haptic feedback, which translates in-game sounds into physical sensations. However, delivering this feedback effectively, especially in fast-paced battle games, is challenging. These environments are characterized by a cacophony of sounds, including weapon fire, explosions, and crucial voice communications, making it difficult to isolate specific target sounds for precise haptic responses. Existing methods often rely on direct game engine integration (limited to specific platforms) or simplistic audio filtering, which can misclassify sounds. The core problem is the lack of a robust, real-time system capable of distinguishing critical game sounds (like weapon fire) from distracting noise (like voice chat) in highly complex audio environments, which is essential for generating accurate and timely haptic feedback.

Methods

The researchers propose a deep learning-based approach for sound event detection (SED) and voice activity detection (VAD) to tackle this problem. Their core methodology involves:

  • Dataset Creation: They curated the "BattleSound" dataset, featuring a large volume of audio recordings from PUBG, meticulously annotated with short (0.5s) intervals. This high-resolution labeling is crucial for distinguishing rapidly occurring sounds.
  • Sound Event Detection (SED): They focus on Weapon Sound Event Detection (WSED) to identify gunshots and explosions.
  • Voice Activity Detection (VAD): They implement Voice Chat Activity Detection (VCAD) to filter out distracting voice communications, preventing misclassification as weapon sounds.
  • Deep Learning Models: They employ mobile-sized Convolutional Neural Networks (CNNs) and Convolutional Recurrent Neural Networks (CRNNs) for real-time processing. These models are trained on two representations of the audio: raw audio signals and mel-spectrograms. Mel-spectrograms, which better capture frequency domain features, are shown to be more effective.
  • Label Resolution Analysis: They investigate the impact of annotation interval length on model performance, demonstrating that shorter intervals (like their 0.5s) significantly improve accuracy compared to longer intervals (2s, 4s, 8s).

Datasets

  • BattleSound (New Benchmark): This is the primary dataset introduced in the paper. It comprises audio clips from PlayerUnknown's Battlegrounds (PUBG), annotated with three categories: VOICE, WEAPON, and MIXTURE. Key characteristics include:
  • High-resolution annotations (0.5s interval).
  • Focus on noisy, real-world game environments.
  • Includes variations in weapon sounds (distance, type) and voice chat (number of speakers, speed).
  • Total duration of approximately 7 hours.
  • Existing Benchmarks (for comparison): The paper references several existing datasets like AudioSet, Freefield1010, ToyADMOS, Chime-home, GunShot, UrbanSound, SINS, ESC, and TUT Acoustic Scene. These are primarily used to highlight the limitations of existing benchmarks in terms of annotation resolution and applicability to noisy game environments.

Key Insights and Ideas

  • High-Resolution Annotation is Critical: The most significant insight is the paramount importance of short annotation intervals (0.5s) for real-time sound event detection in noisy gaming scenarios. Longer intervals lead to "confused frames" where non-target sounds are included, drastically degrading performance.
  • Mel-Spectrograms Outperform Raw Audio: For these tasks, mel-spectrogram representations provide richer features that deep learning models can leverage more effectively than raw audio signals, especially for distinguishing subtle differences between sound types.
  • Mobile-Sized Models are Viable: The study demonstrates that even mobile-sized deep learning models can achieve high accuracy for real-time sound detection, making them suitable for integration into gaming hardware or applications.
  • BattleSound Addresses a Niche: The BattleSound dataset fills a crucial gap by providing a benchmark specifically tailored for the challenges of sound-specific feedback generation in noisy battle games, which existing general-purpose audio datasets do not adequately cover.

Key Results

  • High Accuracy Achieved: The developed models achieved impressive accuracy:
  • Weapon Sound Event Detection (WSED): A 2D CNN model using mel-spectrograms achieved 96.02% accuracy.
  • Voice Chat Activity Detection (VCAD): A 2D CRNN model using mel-spectrograms achieved 96.37% accuracy.
  • Impact of Label Resolution: Downsampling the annotation resolution from 0.5s to 8s led to significant accuracy drops:
  • WSED accuracy decreased by 21.8%.
  • VCAD accuracy decreased by 24.6%.
  • Model Performance: 2D models (using mel-spectrograms) generally outperformed 1D models. For VCAD, CRNN models slightly outperformed CNNs due to their ability to capture temporal dependencies in speech.
  • Multi-Class Performance: A single multi-class model (classifying VOICE, WEAPON, and MIXED) achieved 94.25% accuracy with a 2D CNN and mel-spectrograms.

Limitations and Future Work

  • Limited Scope of BattleSound: While comprehensive for its purpose, BattleSound is derived from a single game (PUBG). Future work could expand this to include a wider variety of game genres and sound types.
  • Real-World Deployment: The paper focuses on model performance. Further research could explore the practical implementation and integration of these models into actual gaming hardware or software for real-time haptic feedback systems.
  • Generalizability: While the models show good performance on BattleSound, their generalizability to other noisy environments or different game soundscapes could be further investigated.
  • Annotation Labor: The manual annotation process, even with short intervals, is labor-intensive. Exploring semi-supervised or self-supervised learning techniques to reduce annotation burden could be a valuable future direction.

Summary updated

BibTeX
@article{shin2023battlesound,
  author = {Sungho Shin and Seongju Lee and Changhyun Jun and Kyoobin Lee},
  title = {{BattleSound: A Game Sound Benchmark for the Sound-Specific Feedback Generation in a Battle Game}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23020770},
}
Figure