Explaining Motion Relevance for Activity Recognition in Video Deep Learning Models
Introduction: The Challenge of Spatial Bias
3D Convolutional Neural Networks (CNNs) have become the standard for video-based activity recognition. However, these models often operate as "black boxes," making it difficult to discern whether they are identifying an action based on actual movement or merely relying on static scene context. For instance, a model might correctly classify a video as "tennis" simply by recognizing a tennis court, a phenomenon known as spatial bias.
While saliency maps—heatmaps indicating influential pixels—are commonly used for 2D image explanations, extending them to 3D video often conflates spatial and temporal information. This lack of granularity prevents researchers from determining if a model is truly learning the dynamics of an activity.
Methodology: Selective Relevance via Temporal Filtering
To address this, the authors propose selective relevance, a post-processing technique designed to isolate motion-specific information. This method acts as a temporal filter for existing explanation frameworks like Deep Taylor Decomposition (DTD).
The process follows three primary steps:
- Baseline Generation: A standard spatio-temporal explanation is generated for the 3D video input.
- Temporal Edge Detection: A Sobel operator is applied along the temporal ($t$) axis of the relevance map. This identifies regions where the model's attention shifts rapidly over time, serving as a proxy for motion.
- Masking: The resulting temporal derivatives are used to create a mask that filters out static background features, leaving only the "motion-relevant" components.
Key Insights: Quantifying Contextual Dependency
A significant finding of this research is the extent of model bias toward static context. The authors quantified that for many baseline explanation methods, approximately 90% of the "relevant" information is actually static spatial context rather than motion. By stripping away this background "noise" (such as stadium seating or landscape features), the resulting explanations become significantly more interpretable for human users.
Furthermore, the study used optical flow as a ground truth for physical movement. The selective relevance method was found to significantly increase the alignment between model explanations and actual pixel movement, suggesting that the filtered maps provide a more honest representation of the model's "motion reasoning."
Results and Performance
The selective relevance approach demonstrates both quantitative and qualitative improvements:
- Precision Gains: For the DTD method, "motion precision" (the overlap with optical flow) improved from 27.79% to 41.09%.
- Efficiency: The method is highly efficient, adding only 0.5 to 1 millisecond of computational overhead, making it suitable for real-time model debugging.
- Qualitative Clarity: Visual comparisons demonstrate that the method successfully isolates the primary actor. In activities like jet skiing or javelin throwing, irrelevant background elements like water spray or the track are removed, focusing the explanation on the athlete's movement.
Limitations and Future Directions
Despite its success, the method faces challenges when paired with low-resolution explanation tools like GradCAM. Because GradCAM produces coarse heatmaps, the temporal changes are often too subtle for the Sobel filter to capture effectively. Additionally, the threshold for motion detection ($\sigma$) currently requires manual tuning per dataset.
Future work may involve applying this technique to modern architectures like D3D, which are specifically designed to emulate optical flow, to further bridge the gap between model explanations and physical reality.
Summary updated