Importance-Guided Interpretability and Pruning for Video Transformers in Driver Action Recognition

Raquel Panadero Palenzuela, Dominik Schörkhuber, Margrit GelautzWACV 2025

TL;DR

This paper tackles the interpretability and computational cost challenges of video transformers for driver action recognition. They introduce a novel layer pruning method guided by head importance metrics, demonstrating significant efficiency gains with minimal accuracy loss, making these powerful models more practical for real-world applications.

What Problem Does This Paper Address?

Video transformers (like ViT and its successors) have become state-of-the-art for action recognition, including critical tasks like driver behavior analysis. However, their complex self-attention mechanisms are often "black boxes," making it hard to understand why a model makes a certain prediction. This lack of interpretability hinders trust, bias identification, and debugging. Furthermore, these models are computationally expensive, especially for video, posing a challenge for deployment in resource-constrained environments like in-car systems. Existing interpretability methods often focus on single images or lack fine-grained analysis of individual attention heads, and pruning techniques haven't extensively explored layer-level optimization for video transformers.

Methods

The authors adapt existing head importance metrics from NLP transformers to video transformers. They use these metrics to quantify the significance of individual attention heads within each layer. Based on these importance scores, they propose a layer pruning technique. This involves training a full video transformer model, then systematically evaluating the importance of each layer by temporarily removing it and measuring the impact on performance (using metrics like probability change and Jensen-Shannon Divergence). Layers deemed "low-relevance" are then removed to create a more efficient model. They also explore pruning "high-relevance" layers for comparison. A key aspect is that this pruning is done after training, allowing for flexible model compression without requiring re-training or fine-tuning.

Datasets

  • DriverActionInsight (DAI): A proprietary dataset specifically for driver action recognition, containing 20 action classes recorded in a controlled environment. It includes 1,302 annotated sequences.
  • UCF101: A widely recognized benchmark dataset for general action recognition, used to demonstrate the generalizability of their approach. It contains 13,320 video sequences across 101 classes.

Key Insights and Ideas

  • Head importance is key to layer importance: By analyzing which attention heads are most critical, the authors can effectively identify which entire layers are less crucial for the model's performance.
  • Low-relevance layers are abundant: The experiments reveal that many layers in video transformers can be removed with minimal impact on accuracy, suggesting significant redundancy.
  • Early layers are critical: High-relevance layers are consistently found at the beginning of the transformer architecture, indicating their foundational role in processing initial visual information.
  • Specialized heads exist: Attention maps show that specific heads within deeper layers focus on very precise visual cues (e.g., a driver's fingers on a phone) and temporal moments, contributing to fine-grained action recognition.
  • Pruning after training is effective: The proposed method allows for efficient model compression without the need for extensive fine-tuning, making it practical for deployment.

Key Results

  • Significant FLOPs reduction: On the DAI dataset, they achieved a 23.5% FLOPs saving for Video Swin with less than a 1% decrease in Top-1 accuracy by pruning low-relevance layers.
  • Pruning high-relevance layers is detrimental: Removing even a single high-relevance layer (which resulted in only a ~5-8% FLOP saving) led to drastic accuracy drops (e.g., ~30% for Video Swin on DAI).
  • Model-specific prunability: Video Swin and MViT showed greater resilience to pruning compared to TimeSformer, suggesting differences in how layers contribute to overall performance.
  • Qualitative validation: Attention visualizations confirmed that low-relevance heads often focus on irrelevant areas, while high-relevance heads capture critical cues for action recognition.

Limitations and Future Work

  • Layer-level impact vs. model-level impact: The authors chose to focus on model-level impact metrics (changes in final output) over layer-level impact metrics (changes in intermediate activations). This was due to the hierarchical nature of models like Video Swin and MViT, where comparing activations across layers with different patch sizes could be problematic.
  • Fine-tuning omitted: For efficiency, fine-tuning after pruning was omitted, though the authors note that minimal performance gains might be achievable with it.
  • Future directions: The authors suggest exploring dynamic adjustment of layers, channels, or heads based on data attributes and dataset complexity. They also propose scaling transformer architectures based on input data characteristics to further improve resource efficiency.

Summary updated

BibTeX
@inproceedings{palenzuela2025importance,
  author = {Raquel Panadero Palenzuela and Dominik Schörkhuber and Margrit Gelautz},
  title = {{Importance-Guided Interpretability and Pruning for Video Transformers in Driver Action Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}

Loading related content...

Figure