What Places an Action Boundary? Structure-Aware Attribution for Temporal Action Localization

Ruoshan Lan, Zongru Shao, Chieh Tsai, Mihai SurdeanuarXiv 2026 · arXiv

In the world of video understanding, Temporal Action Localization (AL) is the task of not just identifying what is happening (classification) but exactly when it starts and ends (localization). While AI models have become quite good at the "what," the "when" remains a major bottleneck.

A new paper, "What Places an Action Boundary? Structure-Aware Attribution for Temporal Action Localization," tackles this by peering under the hood of state-of-the-art TAL detectors to understand how they actually make these temporal decisions.

TL;DR

The authors propose a structure-aware attribution framework that explains TAL predictions by averaging contributions across random evidence-restoration orders. Their findings reveal a striking "class-boundary disconnect": the visual evidence that makes a model confident in an action class is often entirely different from the evidence it uses to place temporal boundaries.

What Problem Does This Paper Address?

Localization errors are the primary source of failure in TAL. To fix them, we need to know what the model is looking at. However, standard attribution methods (like Integrated Gradients) struggle with video for two reasons:

  1. Temporal Redundancy: Neighboring video snippets often carry the same information. If you mask one, the model might still "see" the action through its neighbor, leading to misleadingly low importance scores for individual frames.
  2. Context Dependency: A snippet’s importance changes depending on whether its neighbors are present.

Existing methods often treat localization as a byproduct of classification, but the authors argue that placing a boundary is a distinct task requiring its own specialized explanation logic.

Methods

The authors introduce a framework based on the Shapley–Shubik principle, which averages the marginal contribution of input snippets across many different "restoration orders" (starting from a blank video and adding snippets back in). To make this computationally feasible, they introduced several innovations:

  • Receptive-Field Windowing: Instead of ordering every snippet in a long video, they only independently order snippets within the detector's specific receptive field, treating everything else as a single "context unit."
  • Soft Random-Order Paths: Rather than adding snippets one-by-one (which is slow), they use a continuous gradient path where consecutive restoration transitions overlap, making the cost independent of the number of snippets.
  • Cell-Structured Orders: For boundaries, they group snippets into "cells" based on the detector’s internal feature pyramid. This ensures that snippets that are processed together are restored together.
  • Boundary-Anchored References: They created specialized baselines that selectively remove evidence only near the predicted boundary to see how the start/end times shift.
Figure 1: The framework uses receptive-field windows and soft paths to make Shapley-based attribution tractable for long videos.

Datasets

The framework was validated using two prominent TAL detectors, TriDet and ActionFormer, across two major datasets:

  • THUMOS14: A classic benchmark for action localization in sports and daily activities.
  • EPIC-KITCHENS-100: A challenging dataset of egocentric (first-person) videos of kitchen activities, featuring many short, overlapping actions.

Key Insights and Ideas

  • The Class-Boundary Disconnect: Perhaps the most significant finding is that class-score changes do not reliably track boundary displacement. A model can be 99% sure it's seeing "Cliff Diving" based on the middle of the action, but it uses entirely different, boundary-adjacent evidence to decide exactly when the diver hit the water.
  • Structure Matters: Grouping snippets by the detector's internal "cells" (the temporal resolution at which it makes predictions) significantly improves the faithfulness of the explanations.
  • Directional Faithfulness: The authors argue that localization attribution must be signed. We need to know if a piece of evidence moves a boundary earlier or later, not just that it moves it.
Figure 2: Boundary-specific attribution uses cell-structured orders and anchored references to probe the evidence that specifically shifts the start and end times.

Key Results

  • Improved Faithfulness: The proposed method outperformed existing baselines (like Integrated Gradients and TIMING) on "faithfulness" metrics, which measure how much the model's output actually changes when high-attribution snippets are removed.
  • Efficiency: Their "hybrid" path approach achieved a 4.1x to 12.2x speedup over exact difference methods while maintaining nearly identical attribution quality.
  • Input-Level Evidence: Through controlled interventions (replacing snippets with background), they proved that classification relies on content that is fundamentally less informative about boundary placement.
Figure 3: Visualization showing that class attribution (left) peaks inside the action, while boundary attribution (right) is concentrated at the edges.

Limitations and Future Work

  • Focus on Success: The current study focuses on correctly localized detections. The authors note that a natural next step is applying this to localization failures to diagnose why high-confidence detections often have poor boundaries.
  • Internal Mechanisms: While this work looks at input-level attribution, it doesn't fully explore the internal activations of the model. Future work could combine this with techniques like activation patching to see how evidence flows through the network layers.

This research provides a much-needed toolkit for developers looking to build more precise and interpretable video AI, suggesting that if we want better boundaries, we might need to stop looking at the class score.

Summary updated

Abstract

Temporal action localization (TAL) detectors must identify actions and estimate their temporal boundaries, but localization remains a major source of detection errors. While understanding what determines boundary placement requires faithful attribution of temporal evidence, existing attribution methods have not sufficiently addressed the temporal structure of TAL or the need to tailor attribution to boundary localization. We introduce a structure-aware attribution framework that averages contributions across random evidence-restoration orders following the Shapley-Shubik principle. We make it tractable by using the detector's receptive field to limit which time steps are ordered individually, and by approximating each order with a soft gradient path. For boundary predictions, we further introduce cell-structured restoration orders and boundary-anchored references that selectively remove adjacent evidence. We also adapt faithfulness evaluation to assess the joint effects of contiguous temporal evidence and the direction of boundary displacement. Experiments on THUMOS14 and EPIC-KITCHENS-100 with TriDet and ActionFormer demonstrate improved attribution faithfulness over existing methods. Attribution analysis and controlled interventions further reveal that class-score changes do not reliably track boundary displacement, providing input-level evidence that classification relies on content less informative about boundary placement. Code and evaluation artifacts will be publicly released.
BibTeX
@misc{lan2026what,
  author = {Ruoshan Lan and Zongru Shao and Chieh Tsai and Mihai Surdeanu},
  title = {{What Places an Action Boundary? Structure-Aware Attribution for Temporal Action Localization}},
  year = {2026},
  eprint = {2611.25815},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25815},
}
Figure