Abstract
Localizing actions in video is a core task in computer vision. The weakly supervised temporal localization problem investigates whether this task can be adequately solved with only video-level labels, significantly reducing the amount of expensive and error-prone annotation that is required. A common approach is to train a frame-level classifier where frames with the highest class probability are selected to make a video-level prediction. Frame level activations are then used for localization. However, the absence of frame-level annotations cause the classifier to impart class bias on every frame. To address this, we propose the Action Selection Learning (ASL) approach to capture the general concept of action, a property we refer to as "actionness". Under ASL, the model is trained with a novel class-agnostic task to predict which frames will be selected by the classifier. Empirically, we show that ASL outperforms leading baselines on two popular benchmarks THUMOS-14 and ActivityNet-1.2, with 10.3% and 5.7% relative improvement respectively.
What Problem Does This Paper Address?
Temporal Action Localization (TAL) aims to find the "start" and "end" times of actions in a video. Doing this with weak supervision (video-level labels only) is a major goal because frame-by-frame annotation is incredibly expensive and slow.
Current weakly supervised methods usually use "Multiple Instance Learning" (MIL). They train a classifier to pick the $k$ frames that best represent a class. However, this leads to two major headaches:
- Context Error: The model triggers on "contextual artifacts." For example, it might label a shot of a cricket pitch as a "Cricket Shot" even if no one is actually playing, simply because the pitch is a strong indicator of the sport.
- Actionness Error: The model misses "hard" examples. If a "Cricket Bowling" action happens in an unusual indoor setting or is partially obscured, the classifier might ignore it in favor of easier, clearer frames.
Methods
The authors propose Action Selection Learning (ASL), which uses two complementary networks:
- The Base Classifier ($F$): A standard network that predicts the probability of specific action classes (e.g., "Diving" vs. "High Jump").
- The Actionness Model ($G$): A class-agnostic network that predicts a single score: "Is any action happening here?"
The Selection Task
Instead of just training $G$ on its own, the authors created a novel task: $G$ is trained to predict which frames the classifier $F$ will select for its top-$k$ predictions. Because context (like a stadium or a pool) is highly specific to a class, but the motion of an action is often universal across classes, the actionness model learns to ignore the static background and focus on the movement. The two models are fused using a selection function that combines their beliefs.
To handle the "noise" of early training (where the model might pick the wrong frames initially), they used a Generalized Cross Entropy (GCE) loss. This acts as a noise-tolerant buffer, preventing the model from over-fitting to incorrect early guesses.
Datasets
The model was evaluated on two industry-standard benchmarks for untrimmed video localization:
- THUMOS-14: 200 training and 212 test videos across 20 action classes.
- ActivityNet-1.2: A larger dataset with 4,819 training and 2,383 test videos across 100 action classes.
Key Insights and Ideas
- Actionness is Universal: The core "aha moment" is that while "Diving" and "Hammer Throw" look different, they both share a "property of action" that is distinct from a static background.
- Mutual Improvement: As the classifier gets better at picking frames, the actionness model gets better targets to learn from. Conversely, as the actionness model improves, it filters out "contextual" frames, helping the classifier focus on the actual action.
- Noise Tolerance is Vital: In weak supervision, your "ground truth" for frames is essentially a guess. Using a loss function (GCE) that doesn't heavily penalize "disagreements" early on allows the model to converge on the true action segments more reliably.
Key Results
ASL set new state-of-the-art records for weakly supervised localization at the time of publication:
- THUMOS-14: Achieved 32.2% mAP (mean Average Precision), a 10.3% relative improvement over previous leading methods like DGAM and TSCN.
- ActivityNet-1.2: Achieved 25.8% mAP, a 5.7% relative improvement.
- Qualitative Success: In tests, ASL was much better at "spreading" its predictions across an entire action (like a long hammer throw) rather than just hyper-focusing on the easiest single frame.
Limitations and Future Work
- Initial Noise: The authors acknowledge that early in training, the labels are very noisy, which can still be a hurdle for the model to overcome.
- Complexity: While effective, the dual-network approach adds more parameters and complexity compared to a single-stream MIL model.
- Future Directions: The authors suggest exploring more complex architectures for the "selection function" and testing the "actionness" concept in other video domains beyond sports and daily activities.
Summary updated