Enabling Weakly-Supervised Temporal Action Localization from On-Device Learning of the Video Stream

Yue Tang, Yawen Wu, Peipei Zhou, Jingtong HuarXiv 2022 · arXiv

TL;DR

This paper introduces a framework for on-device temporal action localization (TAL) that learns directly from a continuous, raw video stream. By using a self-adaptive segment merging approach and a novel "interests-based" sampling strategy, the system achieves performance comparable to state-of-the-art weakly-supervised methods without the need for laborious manual video splitting.

What Problem Does This Paper Address?

Temporal Action Localization (TAL) involves identifying both what action is happening in a video and when it starts and ends. While powerful, standard TAL models are typically trained on the cloud using thousands of manually trimmed and annotated video clips.

For real-world on-device applications—like a medical robot monitoring a patient or an autonomous drone—this approach fails for three reasons:

  1. Manual Labor: Real-world video is a continuous stream. Manually cutting this stream into discrete clips for training is prohibitively expensive.
  2. Privacy: Uploading raw video streams to the cloud for retraining compromises user privacy.
  3. Domain Adaptation: A model trained on generic cloud data often needs to adapt to the specific environment of the device (e.g., a specific living room or factory floor).

Existing Weakly-Supervised TAL (W-TAL) methods reduce annotation effort by only requiring video-level labels (e.g., "this 30-second clip contains a fall"), but they still assume the video has already been neatly segmented. This paper bridges the gap by enabling W-TAL to work directly on raw, un-split video streams.

Methods

The authors propose a three-step streaming learning workflow:

  1. Feature Extraction: The continuous stream is initially divided into uniform, non-overlapping segments. A frozen, pre-trained 3D-CNN (I3D) extracts features, which are much smaller to store than raw video.
  2. Interests-Based Sampling (IS): To minimize labeling costs, the system must choose which segments to send to a human "oracle" for labeling. The IS strategy (shown in Interests-based sampling) identifies "areas of interest" where action proposals overlap across different temporal scales, prioritizing segments most likely to contain useful learning signals.
  3. Self-Adaptive Segment Merging (CS-M): Once labels are received, the model needs to know if a segment is too short to contain a full action. The CS-M approach (illustrated in Contrast score-based segment merging) uses a contrast score to measure how well the model can distinguish between "easy action" and "easy background" snippets. If merging two adjacent segments improves this distinction, they are combined into a single training sample.
Interests-based sampling
Fig. 3. Contrast score-based segment merging approach.

The entire framework is built upon CoLA, a recent W-TAL baseline that uses contrastive learning to differentiate actions from background noise without frame-level timestamps.

Datasets

  • THUMOS’14: The primary dataset for evaluation, containing 20 action categories. The authors simulated a continuous stream by concatenating the original videos in various orders (random, class-consecutive, etc.).
  • UCF-101: Used to pre-train the TAL model's classifier before it begins learning from the local stream.
  • Kinetics: Used for the pre-trained I3D feature encoder.

Key Insights and Ideas

  • Contrast as a Metric for Completeness: The most clever insight is using the contrastive loss not just for training, but as a diagnostic tool. If a segment has a high contrast score (meaning action and background look similar to the model), it likely lacks a clear "anchor" for either the action or the background, signaling that it needs to be merged with its neighbor to provide a complete learning example.
  • Overcoming the "Streaming" Bottleneck: By storing only extracted features and a few representative clips (about 2-3% of the original data size), the authors make on-device learning feasible on hardware with limited storage.
  • Smart Sampling vs. Uncertainty: While many active learning methods pick the most "uncertain" samples, this paper shows that for TAL, picking "interesting" samples (those with consistent action proposals) is more effective for building a robust model from scratch.

Key Results

The proposed method was tested on hardware like the NVIDIA Jetson TX2 and a GTX 1050 Ti.

  • Accuracy: On THUMOS’14, the approach achieved a mean Average Precision (mAP) of 37.58%, coming remarkably close to the 40.9% achieved by the SOTA baseline (CoLA) which requires manual video splitting.
  • Efficiency: The system reduced the data required for labeling significantly. For the 11-hour THUMOS dataset, the "oracle" only needed to review roughly 15 minutes of representative clips.
  • Feasibility: Training an epoch takes roughly 16 seconds on a GTX 1050 Ti, making the entire training process possible in just a few hours on-device.
Fig. 5. Comparison of different merging strategies in Situation (1) (seed=0). (a) Training the TAL model from the model pre-trained on the trimmed dataset. (b) Training the TAL model from scratch.
Figure 5 shows that the proposed merging strategies (CS-M) significantly outperform basic uniform splitting (WM) and random merging (RM), approaching the theoretical upper bound of manual splitting.

Limitations and Future Work

  • Temporal Correlation Dependency: The performance of different merging iterations varies depending on how "clumped" actions are in the stream. If actions from the same class are highly consecutive, the merging logic needs to be more aggressive.
  • Frozen Encoder: The current method relies on a frozen I3D encoder. While this saves computation, fine-tuning the encoder on-device could potentially unlock even higher accuracy, albeit at a much higher computational cost.
  • Oracle Availability: The model still assumes a "cloud oracle" (human) can provide labels overnight. Future work could explore fully unsupervised methods to remove the human-in-the-loop entirely.

Summary updated

Abstract

Detecting actions in videos have been widely applied in on-device applications. Practical on-device videos are always untrimmed with both action and background. It is desirable for a model to both recognize the class of action and localize the temporal position where the action happens. Such a task is called temporal action location (TAL), which is always trained on the cloud where multiple untrimmed videos are collected and labeled. It is desirable for a TAL model to continuously and locally learn from new data, which can directly improve the action detection precision while protecting customers' privacy. However, it is non-trivial to train a TAL model, since tremendous video samples with temporal annotations are required. However, annotating videos frame by frame is exorbitantly time-consuming and expensive. Although weakly-supervised TAL (W-TAL) has been proposed to learn from untrimmed videos with only video-level labels, such an approach is also not suitable for on-device learning scenarios. In practical on-device learning applications, data are collected in streaming. Dividing such a long video stream into multiple video segments requires lots of human effort, which hinders the exploration of applying the TAL tasks to realistic on-device learning applications. To enable W-TAL models to learn from a long, untrimmed streaming video, we propose an efficient video learning approach that can directly adapt to new environments. We first propose a self-adaptive video dividing approach with a contrast score-based segment merging approach to convert the video stream into multiple segments. Then, we explore different sampling strategies on the TAL tasks to request as few labels as possible. To the best of our knowledge, we are the first attempt to directly learn from the on-device, long video stream.
BibTeX
@misc{tang2022enabling,
  author = {Yue Tang and Yawen Wu and Peipei Zhou and Jingtong Hu},
  title = {{Enabling Weakly-Supervised Temporal Action Localization from On-Device Learning of the Video Stream}},
  year = {2022},
  eprint = {2208.12673},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.12673},
}
Figure