Soft Video Parsing by Label Distribution Learning

Xin Geng, Miaogen LingAAAI 2017 · DOI

Abstract

Traditional video parsing methods often rely on "hard boundaries," assuming that a video frame belongs exclusively to one semantic category. However, real-world actions involve gradual transitions that are inherently ambiguous. This paper introduces Soft Video Parsing (SP), a framework that leverages Label Distribution Learning (LDL) to model these fuzzy boundaries. By assigning a distribution of labels to video segments rather than a single category, the proposed method captures the "graduality" of action transitions. Combined with a bi-directional sliding window training strategy and Segmental Regular Grammars (SRG), the SP method significantly improves action localization accuracy while drastically reducing computational complexity compared to existing structured models.

Introduction

Video parsing is the task of decomposing a video into a sequence of semantic units, such as identifying the "approach," "jump," and "landing" phases in a high-jump sequence. A major challenge in this field is the presence of "ambiguity zones" at action boundaries where a frame may partially resemble multiple categories.

Most current models ignore this ambiguity, leading to two primary issues: decreased accuracy during transitions and the high computational cost of training models (like Structured SVMs) that attempt to find optimal hard fits. This research addresses these problems by treating ambiguity as a valuable signal rather than noise, using a probabilistic approach to label video segments.

Methodology

The Soft Video Parsing framework is built upon three core components:

1. Label Distribution Learning (LDL)

Instead of a binary classification, LDL assigns a probability distribution across all possible labels for a given segment. For example, a transition frame might be labeled as 70% "Background" and 30% "Approaching." This allows the model to explicitly represent the smooth progression of actions.

Label distributions across a high-jump sequence, illustrating the transition from background to action and back.

2. Bi-directional Sliding Window

To create a robust training set, the authors employ a bi-directional sliding window that moves forward and backward across known action boundaries. This process generates a variety of overlapping segments, each with a calculated label distribution that captures how one action fades into the next. This rich dataset allows the model to learn the nuances of transitions effectively.

Bi-directional sliding window process for generating training segments with varying label distributions.

3. Segmental Regular Grammars (SRG)

The video structure is represented using a grammar-based tree. During inference, dynamic programming finds the most likely path through the video by minimizing the Kullback-Leibler (KL) divergence between predicted distributions and ideal templates. Unlike previous methods, SP uses the efficient L-BFGS optimization algorithm, making training significantly faster.

Key Results and Insights

The researchers evaluated SP on the THUMOS 2014 and MSR-II Action datasets. The results demonstrated substantial improvements over baseline methods like RNMS and SSRG:

  • Accuracy Gains: On THUMOS'14, SP achieved an average Precision (AP) 135% higher than RNMS and 69% higher than SSRG. On the MSR-II dataset, the AP improvement reached 456% over RNMS.
  • Efficiency: The shift from complex structured training to LDL optimization resulted in much lower mathematical complexity, making the framework practical for large-scale datasets.
  • Action-Specific Softness: The study found that different actions require different "ambiguity levels" (denoted as $\lambda$). For instance, fast actions like diving have sharp boundaries (low $\lambda$), while continuous actions like a tennis swing benefit from higher ambiguity modeling (high $\lambda$).

Conclusion and Future Work

Soft Video Parsing proves that modeling boundary ambiguity leads to more accurate and efficient video analysis. While the current method requires manual tuning of the ambiguity parameter ($\lambda$), future work could focus on automatically learning this parameter for different action categories. Additionally, integrating this "soft" logic with modern deep learning features, such as 3D Convolutional Neural Networks, holds the potential for even greater performance in complex, non-linear activity recognition.

Summary updated

BibTeX
@inproceedings{geng2017soft,
  author = {Xin Geng and Miaogen Ling},
  title = {{Soft Video Parsing by Label Distribution Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.10729},
}
Figure