Effectively leveraging Multi-modal Features for Movie Genre Classification

Zhongping Zhang, Yiwen Gu, Bryan A. Plummer, Xin Miao, Jiayi Liu, Huayan Wang

Abstract

Movie genre classification presents a unique challenge due to the extreme length of video content and the subtle semantic differences between genres. We present Movie-CLIP, a framework that enhances classification by integrating language augmentation via automatic speech recognition (ASR) and an efficient shot sampling strategy. By focusing on informative dialogue and diverse visual shots, Movie-CLIP achieves a 6–9% improvement in mean Average Precision (mAP) over state-of-the-art baselines while significantly reducing computational overhead.

Introduction

Classifying a movie's genre is inherently different from standard action recognition. While traditional models treat movies as sequences of short clips, movies are long-form narratives where visual cues alone are often insufficient. For instance, a "Drama" and a "Romance" might share similar visual aesthetics but differ fundamentally in dialogue. Furthermore, processing every frame of a multi-hour film is computationally prohibitive and redundant.

Overview of the Movie-CLIP framework compared to standard video classification.

The proposed Movie-CLIP (Movie genre Classification by Language augmentatIon and shot samPling) addresses these issues by "listening" to the audio and "skimming" the visual content, mimicking human behavior when evaluating a film's genre.

Methodology: The Movie-CLIP Architecture

Movie-CLIP is built upon three core pillars designed to maximize information extraction while maintaining efficiency.

Detailed architecture of the Language Augmentation and Shot Sampling modules.

1. Language Augmentation

To overcome the lack of human-provided subtitles in many datasets, Movie-CLIP utilizes Whisper, a robust ASR system, to generate transcripts. To refine this noisy data, a keyword-aware algorithm filters for high-entropy parts of speech (nouns, pronouns, adjectives) that carry the most genre-specific information (e.g., "battle" vs. "wedding").

2. Shot Sampling

Instead of dense frame sampling, the model segments the video into individual shots. It then selects representative shots across different scenes, ensuring a diverse visual representation of the entire movie without the redundancy of consecutive, nearly identical frames.

3. Multi-modal Adaptive Fusion

The framework extracts features from three distinct modalities:

  • Visual: CLIP-based features from sampled keyframes.
  • Audio: PANNs (Pre-trained Audio Neural Networks) to detect environmental sounds (e.g., explosions, laughter).
  • Text: CLIP text encoder features from the ASR transcripts.

An adaptive fusion layer learns to weigh these modalities dynamically, determining which source is most reliable for a specific genre prediction.

Experimental Results

The model was evaluated on the MovieNet and Condensed Movies benchmarks, yielding several key findings:

  • Performance Gains: Movie-CLIP outperformed established models like SlowFast and UniFormer by 6–9% mAP.
  • Efficiency: The shot sampling strategy allowed for superior performance with a fraction of the computational cost required by dense video encoders.
  • Cross-task Utility: The model's understanding of cinematic structure led to a 1.1% AP improvement in Scene Boundary Detection, demonstrating its robustness beyond simple classification.

Key Insights

  • The Power of "Free" Language: Machine-generated transcripts, even when imperfect, provide critical semantic context that visual features alone cannot capture.
  • Diversity Over Density: Sampling diverse shots from across the film is more effective than processing dense clusters of frames, as it avoids the "redundancy trap."
  • Audio Signatures: Specific sound events (e.g., "children's music" for Family films) serve as strong independent predictors for certain genres.

Conclusion and Future Work

Movie-CLIP demonstrates that effective movie genre classification requires a holistic, multi-modal approach. While the current model achieves state-of-the-art results, future work will explore modeling genre inter-dependencies (e.g., Horror-Thriller correlations) and fine-tuning encoders specifically for the cinematic domain to further improve performance on "long-tail" genres like Biography or History.

Summary updated

BibTeX
@inproceedings{zhang2024effectively,
  author = {Zhongping Zhang and Yiwen Gu and Bryan A. Plummer and Xin Miao and Jiayi Liu and Huayan Wang},
  title = {{Effectively leveraging Multi-modal Features for Movie Genre Classification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=dI3QURMQzq},
}
Figure