TL;DR
This paper introduces a framework for on-device temporal action localization (TAL) that learns directly from a continuous, raw video stream. By using a self-adaptive segment merging approach and a novel "interests-based" sampling strategy, the system achieves performance comparable to state-of-the-art weakly-supervised methods without the need for laborious manual video splitting.
What Problem Does This Paper Address?
Temporal Action Localization (TAL) involves identifying both what action is happening in a video and when it starts and ends. While powerful, standard TAL models are typically trained on the cloud using thousands of manually trimmed and annotated video clips.
For real-world on-device applications—like a medical robot monitoring a patient or an autonomous drone—this approach fails for three reasons:
- Manual Labor: Real-world video is a continuous stream. Manually cutting this stream into discrete clips for training is prohibitively expensive.
- Privacy: Uploading raw video streams to the cloud for retraining compromises user privacy.
- Domain Adaptation: A model trained on generic cloud data often needs to adapt to the specific environment of the device (e.g., a specific living room or factory floor).
Existing Weakly-Supervised TAL (W-TAL) methods reduce annotation effort by only requiring video-level labels (e.g., "this 30-second clip contains a fall"), but they still assume the video has already been neatly segmented. This paper bridges the gap by enabling W-TAL to work directly on raw, un-split video streams.
Methods
The authors propose a three-step streaming learning workflow:
- Feature Extraction: The continuous stream is initially divided into uniform, non-overlapping segments. A frozen, pre-trained 3D-CNN (I3D) extracts features, which are much smaller to store than raw video.
- Interests-Based Sampling (IS): To minimize labeling costs, the system must choose which segments to send to a human "oracle" for labeling. The IS strategy (shown in Interests-based sampling) identifies "areas of interest" where action proposals overlap across different temporal scales, prioritizing segments most likely to contain useful learning signals.
- Self-Adaptive Segment Merging (CS-M): Once labels are received, the model needs to know if a segment is too short to contain a full action. The CS-M approach (illustrated in Contrast score-based segment merging) uses a contrast score to measure how well the model can distinguish between "easy action" and "easy background" snippets. If merging two adjacent segments improves this distinction, they are combined into a single training sample.
The entire framework is built upon CoLA, a recent W-TAL baseline that uses contrastive learning to differentiate actions from background noise without frame-level timestamps.
Datasets
- THUMOS’14: The primary dataset for evaluation, containing 20 action categories. The authors simulated a continuous stream by concatenating the original videos in various orders (random, class-consecutive, etc.).
- UCF-101: Used to pre-train the TAL model's classifier before it begins learning from the local stream.
- Kinetics: Used for the pre-trained I3D feature encoder.
Key Insights and Ideas
- Contrast as a Metric for Completeness: The most clever insight is using the contrastive loss not just for training, but as a diagnostic tool. If a segment has a high contrast score (meaning action and background look similar to the model), it likely lacks a clear "anchor" for either the action or the background, signaling that it needs to be merged with its neighbor to provide a complete learning example.
- Overcoming the "Streaming" Bottleneck: By storing only extracted features and a few representative clips (about 2-3% of the original data size), the authors make on-device learning feasible on hardware with limited storage.
- Smart Sampling vs. Uncertainty: While many active learning methods pick the most "uncertain" samples, this paper shows that for TAL, picking "interesting" samples (those with consistent action proposals) is more effective for building a robust model from scratch.
Key Results
The proposed method was tested on hardware like the NVIDIA Jetson TX2 and a GTX 1050 Ti.
- Accuracy: On THUMOS’14, the approach achieved a mean Average Precision (mAP) of 37.58%, coming remarkably close to the 40.9% achieved by the SOTA baseline (CoLA) which requires manual video splitting.
- Efficiency: The system reduced the data required for labeling significantly. For the 11-hour THUMOS dataset, the "oracle" only needed to review roughly 15 minutes of representative clips.
- Feasibility: Training an epoch takes roughly 16 seconds on a GTX 1050 Ti, making the entire training process possible in just a few hours on-device.
Limitations and Future Work
- Temporal Correlation Dependency: The performance of different merging iterations varies depending on how "clumped" actions are in the stream. If actions from the same class are highly consecutive, the merging logic needs to be more aggressive.
- Frozen Encoder: The current method relies on a frozen I3D encoder. While this saves computation, fine-tuning the encoder on-device could potentially unlock even higher accuracy, albeit at a much higher computational cost.
- Oracle Availability: The model still assumes a "cloud oracle" (human) can provide labels overnight. Future work could explore fully unsupervised methods to remove the human-in-the-loop entirely.
Summary updated