Abstract
Video classification using 3D Convolutional Neural Networks (CNNs) often relies on "patch-based learning," where models are trained on short temporal snippets rather than full-length videos. This paper investigates the inherent biases introduced by this training strategy using Deep Taylor Decomposition (DTD). Our analysis reveals two distinct phenomena: the "border effect," characterized by an over-reliance on the first and last frames of a snippet, and the "lookahead effect," where models learn to ignore initial video segments. We show that these explainability-driven insights allow for zero-cost performance improvements through optimized frame sampling.
Introduction
The computational intensity of video processing necessitates training deep neural networks on short "snippets" or patches. While efficient, this approach assumes that class-relevant information is uniformly distributed and locally present—an assumption that often fails in practice. This study aims to decode the "black box" of snippet-trained models to understand how this training paradigm distorts a network's temporal focus and how these distortions can be mitigated.
Methodology: Deep Taylor Decomposition
To visualize the network's decision-making process, we employ Deep Taylor Decomposition (DTD), a technique related to Layer-wise Relevance Propagation (LRP). DTD backpropagates the model's final prediction score through the layers to assign "relevance scores" to input pixels.
Unlike standard gradient-based Sensitivity Analysis, which often produces noisy heatmaps, DTD provides a cleaner representation of the features the model considers important. We evaluated these explanations on a C3D network pre-trained on the Sports-1M dataset, comparing the results against both Sensitivity Analysis and an untrained baseline to ensure the observed patterns were learned features rather than architectural artifacts.
Identifying Temporal Biases
Our analysis identified two primary biases in the model's temporal attention:
- The Border Effect: Quantitative analysis using quadratic regression confirmed that the network disproportionately focuses on the first and last frames of a 16-frame snippet. This "U-shaped" relevance distribution suggests the model is attempting to "look beyond" its limited temporal window to find context missing from the short patch.
- The Lookahead Effect: The model learns to assign significantly less relevance to the very beginning of a video file. This likely reflects a learned adaptation to common video structures (e.g., YouTube intros or logos) that contain no action-relevant information.
Explainability-Driven Optimization
Beyond providing diagnostic value, these insights offer a mechanism for performance tuning. By observing the border effect, we hypothesized that increasing the temporal context within a snippet could improve accuracy.
By simply increasing the frame sampling step size from 1 to 2 during the inference phase—effectively doubling the temporal span of a 16-frame snippet—we reduced the border effect's impact. This minor adjustment to the data pipeline improved top-5 classification accuracy on the Sports-1M dataset by approximately 1% (from a 60% baseline) without requiring any model retraining.
Conclusion
This work demonstrates that post-hoc explainability methods like DTD are powerful tools for identifying systematic biases in machine learning models. By uncovering the "border" and "lookahead" effects, we show that understanding why a model predicts can lead directly to actionable improvements in how it performs, paving the way for more efficient hyperparameter optimization in video and time-series analysis.
Summary updated