Latent Bi-Constraint SVM for Video-Based Object Recognition

Yang Liu, Minh Hoai, Mang Shao, Tae-Kyun KimIEEE TCSVT 2016 · OpenReview

TL;DR

This paper introduces Latent Bi-constraint SVM (LBSVM), a novel framework for video-based object recognition that addresses the challenges of noisy video data and temporal inconsistency. LBSVM achieves state-of-the-art results by leveraging two new constraints and a latent variable for view selection, outperforming existing methods on two newly proposed datasets.

What Problem Does This Paper Address?

Recognizing objects from images is a well-studied problem, but extending this to videos presents unique challenges. Videos often contain noisy frames due to occlusion, motion blur, and varying illumination, making frame-by-frame recognition unreliable. Furthermore, ensuring consistent object recognition decisions across an entire video, despite these variations, is crucial. Existing methods often treat videos as a collection of independent frames or rely on techniques that don't fully capture the temporal dynamics and inherent structure of video data. This paper aims to bridge this gap by developing a robust video-based object recognition system that can handle these complexities.

Methods

The core contribution is the Latent Bi-constraint SVM (LBSVM), a maximum-margin framework built upon Structured-Output SVM. LBSVM introduces two key novel constraints:

  1. Expanded Training and Subsequence Classification: This constraint treats all subsequences of a training video as positive examples for the object. This maximizes the usage of training data and helps the model learn to recognize objects from various viewpoints.
  2. Monotonicity Constraint: This constraint ensures that the recognition score for a video segment is not lower than the score for its subsequences. This enforces temporal consistency, reflecting the intuition that a longer video segment (with more views) should have higher confidence in recognizing the object.

Additionally, LBSVM incorporates a latent variable that acts as a view selector. This allows the model to filter out "bad" frames (e.g., those with occlusion or motion blur) that could negatively impact recognition, further improving robustness and consistency. The optimization problem is solved using a Non-convex Regularized Bundle Method (NRBM).

Datasets

The paper introduces two new datasets for video-based object recognition:

  • Office Dataset: Contains 210 videos of 10 object categories (e.g., mouse, keyboard, fan) captured in a cluttered office environment. This dataset is challenging due to heavy clutter, scale variations, illumination changes, and view shifting.
  • Museum Dataset: Consists of 820 videos of 20 sculptures, characterized by low texture, similar appearances, and challenging capture conditions (crowded environments, varying distances, and inexperienced users).

These datasets are designed to evaluate video-level recognition, requiring a large number of videos for training and testing.

Key Insights and Ideas

  • Video as a Structured Entity: The paper emphasizes that videos are not just collections of frames but have inherent temporal structure and redundancy that can be exploited for better recognition.
  • The Power of Constraints: The two novel constraints are crucial for enforcing temporal consistency and leveraging diverse views within a video, leading to more robust recognition.
  • Latent Variables for Robustness: Using latent variables to select "good" views is a powerful mechanism to mitigate the impact of noisy or uninformative frames, a common problem in real-world videos.
  • Bridging Image and Video Recognition: LBSVM effectively bridges the gap between image-based recognition and the more complex demands of video analysis.

Key Results

LBSVM demonstrates superior performance on both the Office and Museum datasets compared to several baseline methods, including:

  • Average Frame-based Recognition (Avg-Frame): Achieves accuracies of 65.1% (Office) and 67.3% (Museum1).
  • Accumulating Frame Recognition (Accum-Frame): Improves upon Avg-Frame, reaching 74.5% (Office) and 91.5% (Museum1).
  • Best Template Descriptor (BTD): Shows slight improvement over Accum-Frame.
  • Kernel Principal Angles (KPA): Achieves higher accuracy, reaching 79.5% (Office) and 95.0% (Museum1).

LBSVM (proposed) consistently achieves the highest accuracies: 84.5% on the Office dataset, 98.8% on Museum1, and 91.5% on Museum2. The paper also shows that LBSVM's recognition accuracy increases monotonically with the length of the video subsequence, indicating effective information accumulation, unlike other methods that fluctuate.

Limitations and Future Work

  • Computational Cost: While LBSVM is shown to be efficient in testing (11ms per video), training can be computationally intensive, taking around 2 hours for the Office dataset with a maximum of 300 iterations.
  • Parameter Tuning: Like most machine learning models, LBSVM's performance is dependent on parameter settings, which were carefully tuned for the experiments.
  • Future Work: The paper implicitly suggests that further exploration of latent variable models and more sophisticated feature representations could lead to even greater improvements. The introduction of new, challenging datasets also opens avenues for future research in video-based object recognition.

Summary updated

Abstract

We address the task of recognizing objects from video input. This important problem is relatively unexplored, compared with image-based object recognition. To this end, we make the following contributions. First, we introduce two comprehensive data sets for video-based object recognition. Second, we propose latent bi-constraint SVM (LBSVM), a maximum-margin framework for video-based object recognition. LBSVM is based on structured-output SVM, but extends it to handle noisy video data and ensure consistency of the output decision throughout time. We apply LBSVM to recognize office objects and museum sculptures, and we demonstrate its benefits over image-based, set-based, and other video-based object recognition.
BibTeX
@article{liu2016latent,
  author = {Yang Liu and Minh Hoai and Mang Shao and Tae-Kyun Kim},
  title = {{Latent Bi-Constraint SVM for Video-Based Object Recognition}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2016},
  url = {https://openreview.net/forum?id=FvvgiipbPb},
}

Loading related content...

Figure