TL;DR
This paper introduces a "segment-level" supervision setting for temporal action localization, where annotators label only one or two short segments of an action rather than precise start and end times. By combining this efficient annotation with a novel localization module featuring Sphere Loss and Propagation Loss, the authors achieve performance that significantly outperforms video-level weak supervision while requiring nearly the same amount of human effort.
What Problem Does This Paper Address?
Temporal Action Localization (TAL) is the task of identifying both what is happening in a video and when it starts and ends.
- Full Supervision (providing exact timestamps) yields high accuracy but is incredibly expensive and slow to annotate.
- Video-level Weak Supervision (providing only a category tag for the whole video) is cheap but often fails because models tend to focus only on the most "discriminative" parts—the most obvious moments—rather than the full extent of the action. For example, a model might only recognize a "long jump" during the flight phase, completely ignoring the crucial run-up.
The authors propose a middle ground: segment-level supervision. Annotators simply identify a few seconds where the action occurs. This provides a stronger signal than a video-level tag without the high cost of frame-accurate boundary marking.
Methods
The core of the paper is a novel architecture consisting of a Classification Module and a Localization Module.
The Localization Module uses three specific loss functions to bridge the gap between sparse labels and full action detection:
- Partial Segment Loss (PSL): This forces the model to correctly identify the action at the specific segments labeled by the annotator.
- Sphere Loss (SL): Adapted from face recognition, this loss ensures that features belonging to the same action category are tightly clustered in an angular space. This makes it easier for the model to distinguish between similar actions (e.g., "Long Jump" vs. "High Jump").
- Propagation Loss (PL): This is a regularization term that uses a similarity matrix to "spread" the labels from the few annotated segments to other, unannotated segments that look similar in feature space. This helps the model capture the full duration of an action.
Datasets
The researchers evaluated their approach on two major benchmarks:
- THUMOS14: A challenging dataset with 20 action classes in untrimmed videos.
- ActivityNet (v1.2 and v1.3): Large-scale datasets containing hundreds of action classes.
To simulate segment-level labels for these existing datasets, the authors randomly sampled from the available full boundary annotations. They also conducted a user study using the COIN annotation tool to measure the actual time cost of their proposed labeling method.
Key Insights and Ideas
- Breaking the "Discriminative" Trap: By providing even a single labeled segment, the model is forced to look beyond the most obvious frames. Figure 4 illustrates how each loss term helps the model expand its prediction from a tiny "discriminative" sliver to the full action extent.
- Similarity as a Bridge: The Propagation Loss relies on the intuition that segments belonging to the same action instance will have similar features. By penalizing differences between similar segments, the model naturally learns to group them together into a single action proposal.
- Efficiency vs. Accuracy: The authors demonstrate that the "annotation-duration ratio" (the time spent labeling vs. the video length) for segment-level supervision is only marginally higher than video-level labeling, yet it yields a massive jump in localization accuracy.
Key Results
- Performance Boost: On THUMOS14 (at IoU=0.5), the proposed method achieved 31.6% mAP, significantly outperforming state-of-the-art video-level methods like BaSNet (27.0%).
- Competitive with Full Supervision: On ActivityNet 1.2, the method reached 26.4% average mAP, which is remarkably close to the fully supervised SSN model (26.6%), despite using much simpler annotations.
- Annotation Efficiency: The time cost for "two-segment" labeling was measured at a ratio of 0.32, compared to 0.24 for video-level and a staggering 1.17 for full supervision. This proves that segment-level labels offer the best "bang for your buck."
Limitations and Future Work
While the propagation loss is powerful, it relies on the assumption that features remain similar throughout an action. If an action changes drastically in appearance from start to finish (e.g., a complex gymnastics routine), the propagation might struggle to connect the segments.
The authors suggest that future work could involve more sophisticated feature extraction or exploring how to optimally select which segments an annotator should label to provide the most information to the model.
Summary updated