Mid-level Representation for Visual Recognition

Moin NabiarXiv 2015 · OpenReview

Abstract

Visual Recognition is one of the fundamental challenges in AI, where the goal is to understand the semantics of visual data. Employing mid-level representation, in particular, shifted the paradigm in visual recognition. The mid-level image/video representation involves discovering and training a set of mid-level visual patterns (e.g., parts and attributes) and represent a given image/video utilizing them. The mid-level patterns can be extracted from images and videos using the motion and appearance information of visual phenomenas. This thesis targets employing mid-level representations for different high-level visual recognition tasks, namely (i)image understanding and (ii)video understanding. In the case of image understanding, we focus on object detection/recognition task. We investigate on discovering and learning a set of mid-level patches to be used for representing the images of an object category. We specifically employ the discriminative patches in a subcategory-aware webly-supervised fashion. We, additionally, study the outcomes provided by employing the subcategory-based models for undoing dataset bias.

Introduction

In computer vision, a significant "semantic gap" exists between low-level image features (such as pixels or edge descriptors) and high-level semantic labels. This gap presents several challenges: high intra-class variation, where objects of the same category appear differently; dataset bias, where models fail to generalize across different environments; and supervision bottlenecks caused by the high cost of manual labeling. This research addresses these issues by developing mid-level representations—such as discriminative image patches and temporal poselets—that are descriptive enough to capture semantics but simple enough to be learned with minimal or "webly" supervision.

Image Understanding: Subcategories and Bias Mitigation

The first part of this research focuses on object detection and recognition through subcategory-aware modeling. Instead of a single monolithic classifier, the proposed Latent Subcategory Models (LSM) use a mixture of models for visually consistent subcategories (e.g., "frontal car" vs. "side-view car").

To combat dataset bias, a multitask learning framework is introduced that treats multiple biased datasets as related tasks. By learning a shared "visual world" model alongside dataset-specific bias terms, the system can leverage statistical strength from diverse sources while filtering out unique dataset quirks. Furthermore, the research utilizes the LEVAN (Learning Everything about ANything) framework to mine discriminative, fixed-position patches from millions of web images. These simple patches effectively replace complex deformable part models by leveraging the visual consistency of subcategories.

Video Understanding: Crowd Dynamics and Activity Recognition

The second part of the thesis extends mid-level representations to video data. For crowd analysis, a tracklet-based "commotion measure" is proposed. This unsupervised method calculates how local motion patterns deviate from the dominant motion in a scene, enabling real-time abnormality detection without prior training on specific behaviors.

For group activity recognition, the research introduces Temporal Poselets (TPOS). By extending the concept of poselets (human part detectors) into the time domain, TPOS captures the dynamics of human interaction. The descriptor analyzes the correlation of poselet activations over a temporal window, providing a robust proxy for behavior even in scenes with significant background clutter or camera motion.

Experimental Results

The proposed frameworks were evaluated on standard benchmarks including PASCAL VOC 2007, the UCSD crowd dataset, and the Collective Activity Datasets (CAD).

  • Bias Removal: The subcategory-based bias removal method showed a ~8.5% improvement over standard aggregated models in cross-domain testing.
  • Object Detection: The webly-supervised patch model achieved 19.5% Average Precision (AP) on the "mountain horse" subcategory, significantly outperforming the LEVAN baseline of 15.9%.
  • Crowd Analysis: The commotion measure achieved an AUC of 0.9889 on the UMN dataset, matching or beating state-of-the-art supervised methods while running at a real-time 5 fps.
  • Activity Recognition: The TPOS descriptor improved group activity classification by 10.1% on CAD2 and 19.5% on CAD3 compared to standard feature-level baselines.

Conclusion and Future Work

Mid-level representations offer a powerful bridge across the semantic gap in visual recognition. By explicitly modeling subcategories and dataset bias, and by extending part-based descriptors into the temporal domain, this research provides robust tools for understanding complex visual scenes. Future work will explore adaptive subcategorization to learn the optimal number of sub-groups automatically and investigate richer patch models for more diverse motion representation.

Summary updated

Abstract

Visual Recognition is one of the fundamental challenges in AI, where the goal is to understand the semantics of visual data. Employing mid-level representation, in particular, shifted the paradigm in visual recognition. The mid-level image/video representation involves discovering and training a set of mid-level visual patterns (e.g., parts and attributes) and represent a given image/video utilizing them. The mid-level patterns can be extracted from images and videos using the motion and appearance information of visual phenomenas. This thesis targets employing mid-level representations for different high-level visual recognition tasks, namely (i)image understanding and (ii)video understanding. In the case of image understanding, we focus on object detection/recognition task. We investigate on discovering and learning a set of mid-level patches to be used for representing the images of an object category. We specifically employ the discriminative patches in a subcategory-aware webly-supervised fashion. We, additionally, study the outcomes provided by employing the subcategory-based models for undoing dataset bias.
BibTeX
@misc{nabi2015mid,
  author = {Moin Nabi},
  title = {{Mid-level Representation for Visual Recognition}},
  year = {2015},
  eprint = {1512.07314},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1512.07314},
}
Figure