Analysis of Environmental Sounds as Indexical Signs in Film

Simon Moncrieff, Chitra Dorai, Svetha VenkateshIEEE Pacific Rim Conference on Multimedia 2001 · DOI

TL;DR

This paper introduces a system for classifying specific environmental sounds in film audio that act as "indexical signs" – sounds that directly signify events like car chases or violence. By developing novel audio features and testing them on both pure sounds and actual movie scenes, they demonstrate a promising step towards automatically understanding the thematic content of film audio.

What Problem Does This Paper Address?

While film audio is crucial for enhancing viewer experience, automatically understanding its content beyond dialogue remains a challenge. Specifically, this paper tackles the problem of identifying and classifying non-literal sound effects that serve as "indexical signs." These are sounds that, by their very nature, point to specific events or actions within a scene, such as the screech of tires for a car chase or gunshots for violence. The goal is to move beyond basic audio segmentation and towards a higher-level understanding of scene content and dramatic purpose, enabling more sophisticated media indexing and analysis.

Methods

The authors propose a classification system that utilizes a combination of established and novel audio features to detect ten specific environmental sound events. These events are chosen because they are characteristic of car chase and violent scenes.

  • Feature Engineering: They extract a comprehensive set of audio features, including low-level psychoacoustic and physical features, as well as higher-order features derived from them. A key contribution is the development of new audio features designed to capture specific aspects of sound events:
  • Boundary Behaviour (Loudness Minima and Maxima): Measures the number of peaks and troughs in the raw and smoothed loudness of a signal, indicating the dynamic nature of a sound.
  • Extrema: Quantifies the proportion of the smoothed energy envelope that falls into high or low energy regions, helping to distinguish sounds with sustained energy from those with rapid attacks and decays.
  • Repetition: A binary feature indicating whether a sound event occurs multiple times within a sample, useful for differentiating single events from bursts (e.g., single gunshot vs. automatic gunfire).
  • Inharmonicity: Measures the non-harmonic content of a signal, distinguishing sounds with clear tonal components from those with more noise-like characteristics.
  • Harmonicity: Assesses the stable harmonic content of a sound, looking at the consistency of the fundamental frequency over time.
  • Classification: The extracted features are used to train classifiers. They experiment with Decision Trees (DT) and Support Vector Machines (SVM). An "optimal feature set" of 33 features was determined through a process of refining the initial large set based on classification efficacy and discrimination power.

Datasets

  • Pure Sound Samples: A dataset of 861 homogeneous sound samples, covering the ten event categories, was collected from the WWW. These were converted to 16-bit mono at a 22050Hz sampling rate.
  • Movie Audio Tracks: Sample scenes were extracted from two action movies: Lethal Weapon 2 and The Mummy. These tracks contain mixed sounds, representing real-world film audio.

Key Insights and Ideas

  • Indexical Signs as a Foundation: The paper highlights the importance of "indexical signs" – sounds that directly point to events – as a crucial building block for understanding complex film audio scenes.
  • Novel Features Enhance Discrimination: The newly developed audio features, particularly those related to boundary behavior, repetition, and harmonicity, proved effective in distinguishing between different sound events, offering better discrimination than traditional features alone.
  • Context Matters: While pure sound classification achieved high accuracy, performance dropped when applied to actual movie scenes. This underscores the challenge of dealing with mixed audio and overlapping sound events in real-world film soundtracks.
  • Misclassifications Reveal Scene Structure: Even when misclassified, the errors often occurred between related sound events within the same high-level scene category (e.g., car chase sounds being confused with other car chase sounds). This suggests the system can still provide valuable information about the broader scene type.

Key Results

  • Pure Sound Classification: The Decision Tree classifier achieved an average of 88.9% correct classification on pure sound samples across 25 test trials. SVMs achieved 87.7% using five-fold cross-validation.
  • Movie Audio Track Classification:
    • The Decision Tree correctly recognized 93.8% of car chase events and 73.0% of violence events.
  • SVMs achieved 80.4% for car chase events and 78.2% for violence events.
  • Car Crash Events: These were noted as being particularly difficult to classify, likely due to their less structured nature.
  • Low-Level Sound Interference: The presence of background engine noise in car chase scenes led to misclassifications of sounds like horns and sirens as engine sounds.

Limitations and Future Work

  • Mixed Audio Challenges: The paper acknowledges that the classification accuracy on real film audio is lower than on pure sounds due to the complexity of mixed audio tracks.
  • Semantic Labeling Issues: Misclassifications for multiple gunfire events were partly attributed to the semantic labeling of certain samples in The Mummy, where multiple simultaneous gunshots were labeled as "multiple" or "automatic" gunfire.
  • Future Directions: The authors suggest that reliable sound event detection is the "first step" towards investigating the feasibility of sound scene detection in film, which would further aid in deducing thematic scene content. This implies a future focus on combining these detected events into higher-level scene interpretations.

Summary updated

BibTeX
@inproceedings{moncrieff2001analysis,
  author = {Simon Moncrieff and Chitra Dorai and Svetha Venkatesh},
  title = {{Analysis of Environmental Sounds as Indexical Signs in Film}},
  booktitle = {IEEE Pacific Rim Conference on Multimedia},
  year = {2001},
  doi = {10.1007/3-540-45453-5\_69},
}
Figure