Fine-Grained Engine Fault Sound Event Detection Using Multimodal Signals

Dennis Fedorishin, Livio Forte, Philip Schneider, Srirangaraj Setlur, Venu Govindaraju

Introduction

Automated engine diagnostics have traditionally relied on sample-level classification, which identifies the presence of a fault within a broad recording but fails to pinpoint its exact timing. However, effective mechanical diagnosis requires understanding the precise temporal context of a fault—such as whether a metallic knock occurs during startup, idling, or revving.

This paper addresses the challenge of Sound Event Detection (SED) in the automotive domain: identifying the exact onset and offset times of specific mechanical anomalies. Achieving high-resolution temporal detection is significantly more difficult than simple classification, particularly due to the scarcity of "strongly-labeled" datasets where every second of audio and vibration is precisely annotated.

Methodology: Multimodal CRNN and Contrastive Learning

The researchers propose a multimodal Convolutional Recurrent Neural Network (CRNN) architecture designed to process both acoustic and kinetic signals.

Feature Extraction and Fusion

The model utilizes two distinct input streams:

  1. Audio: Log-Mel spectrograms captured via professional-grade microphones.
  2. Vibration: Linear magnitude spectrograms derived from tri-axial accelerometers.

Each modality is processed through a dedicated 2D CNN to extract high-level features. Because these signals may have different sampling characteristics, the vibration features are interpolated to match the temporal resolution of the audio features before being concatenated into a fused representation.

Overview of the multimodal CRNN architecture (a) and the contrastive pretraining strategy (b).

Temporal Modeling and Pretraining

A bi-directional Gated Recurrent Unit (Bi-GRU) analyzes the fused features to capture long-term temporal dependencies, allowing the model to understand how sound events evolve over time. To mitigate the limitations of a small strongly-labeled dataset, the authors implemented a novel pretraining strategy using over 100,000 weakly-labeled samples. They employed a Supervised Contrastive Loss, which encourages the model to learn similar internal representations for engines exhibiting the same faults, thereby improving feature robustness before final fine-tuning.

Datasets

The study utilizes two primary data sources:

  • Fine-Grained SED Dataset: A newly collected dataset comprising 2,643 audio-vibration samples from 232 vehicle models. It includes over 5,000 expertly labeled individual sound events across 10 categories, such as Engine Knock, Belt Squeal, and Startup Rattle.
  • Large-Scale Pretraining Dataset: A dataset of over 100,000 recordings with weak labels (indicating fault presence without timing) used to initialize the model weights.

Results and Discussion

The integration of multimodal data and contrastive pretraining yielded significant performance improvements. The multimodal approach achieved a 2.5% increase in the Polyphonic Sound Detection Score (PSDS) and a 1.5% boost in classification accuracy compared to single-modality models.

Qualitative comparison of sound event detection performance between audio-only and multimodal pretrained models, highlighting improvements in detecting 'Trouble Starting' and 'Unstable Idle'.

Key Insights

  • Vibration as a Critical Modality: While audio is effective for most faults, vibration data is essential for detecting "Unstable Idle," which often manifests as physical shaking rather than a distinct acoustic signature.
  • Impact of Contrastive Loss: Supervised contrastive pretraining outperformed standard pretraining by approximately 4.5% on PSDS scores, helping the model distinguish between acoustically similar faults by focusing on underlying mechanical characteristics.
  • Temporal Context: The results confirm that the timing of a sound relative to engine states (e.g., startup vs. idle) is a vital diagnostic clue that sample-level classification typically misses.

Limitations and Future Work

Despite these gains, certain faults remain challenging to differentiate. "Engine Knock" and "Internal Tick" are frequently confused due to their similar acoustic profiles and physical origins. Furthermore, the "polyphonic" nature of engine noise—where multiple faults overlap—remains a complex hurdle. Future research may explore Transformer-based architectures to better model these overlapping events and further improve temporal precision.

Summary updated

Abstract

Sound event detection (SED) is an active area of audio research that aims to detect the temporal occurrence of sounds. In this paper, we apply SED to engine fault detection by introducing a multimodal SED framework that detects fine-grained engine faults of automobile engines using audio and accelerometer-recorded vibration. We first introduce the problem of engine fault SED on a dataset collected from a large variety of vehicles with expertly-labeled engine fault sound events. Next, we propose a SED model to temporally detect ten fine-grained engine faults that occur within vehicle engines and further explore a pretraining strategy using a large-scale weakly-labeled engine fault dataset. Through multiple evaluations, we show our proposed framework is able to effectively detect engine fault sound events. Finally, we investigate the interaction and characteristics of each modality and show that fusing features from audio and vibration improves overall engine fault SED capabilities.
BibTeX
@inproceedings{fedorishin2024fine,
  author = {Dennis Fedorishin and Livio Forte and Philip Schneider and Srirangaraj Setlur and Venu Govindaraju},
  title = {{Fine-Grained Engine Fault Sound Event Detection Using Multimodal Signals}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=QAjvp7MYqK},
}
Figure