Deformable Part Descriptors for Fine-Grained Recognition and Attribute Prediction

Ning Zhang, Ryan Farrell, Forrest N. Iandola, Trevor DarrellICCV 2013 · OpenReview

Abstract

Recognizing objects in fine-grained domains can be extremely challenging due to the subtle differences between subcategories. Discriminative markings are often highly localized, leading traditional object recognition approaches to struggle with the large pose variation often present in these domains. Pose-normalization seeks to align training exemplars, either piecewise by part or globally for the whole object, effectively factoring out differences in pose and in viewing angle. Prior approaches relied on computationally-expensive filter ensembles for part localization and required extensive supervision. This paper proposes two pose-normalized descriptors based on computationally-efficient deformable part models. The first leverages the semantics inherent in strongly-supervised DPM parts. The second exploits weak semantic annotations to learn cross-component correspondences, computing pose-normalized descriptors from the latent parts of a weakly-supervised DPM. These representations enable pooling across pose and viewpoint, in turn facilitating tasks such as fine-grained recognition and attribute prediction. Experiments conducted on the Caltech-UCSD Birds 200 dataset and Berkeley Human Attribute dataset demonstrate significant improvements of our approach over state-of-art algorithms.

Introduction: The Challenge of Fine-Grained Recognition

Recognizing "fine-grained" categories—such as distinguishing a Lincoln Sparrow from a Song Sparrow—is significantly harder than general object recognition. While an elephant is easily identified by the presence of a trunk, sub-species are often differentiated by subtle, localized variations in the shape or color of specific parts, such as the pattern on a bird's wing or the shape of its beak.

The primary challenge is pose variation. An object's appearance changes drastically depending on its orientation and articulation. Traditional "bag-of-words" models struggle because they do not account for where features are located. Previous attempts at "pose-normalization" (aligning parts to a common template) relied on "Poselets," which are computationally expensive to run and require massive amounts of manual supervision to train.

Methodology: Deformable Part Descriptors (DPD)

The authors introduce Deformable Part Descriptors (DPD), a framework that uses Deformable Part Models (DPM) to achieve efficient pose-normalization. The core insight is that the internal structure of a DPM—originally designed for detection—is an excellent, lightweight tool for semantic alignment.

Overview of the DPD framework for fine-grained recognition and attribute prediction.

The framework offers two versions of the descriptor:

  1. DPD-Strong: Uses a DPM trained with explicit semantic part annotations (e.g., "head" and "body"). The model learns to find these specific parts, and features are extracted directly from the predicted part boxes.
  2. DPD-Weak: Uses a standard DPM where parts are "latent." To make these parts semantically meaningful, the authors use a small set of keypoint annotations to learn a mapping—a "pooling weight"—that tells the system which latent parts correspond to which semantic regions (like the head, torso, or legs).

Once parts are localized, the system extracts kernel descriptors (capturing gradient, texture, and color information) and pools them into a "pose-normalized" representation. This allows a classifier to compare the "head" of a bird in one image to the "head" of a bird in another, regardless of orientation.

Evaluation and Results

The researchers evaluated DPD on three major benchmarks: the Caltech-UCSD Birds (CUB200-2010 and CUB200-2011) datasets and the Berkeley Human Attribute Dataset.

Examples of part localization and segmentation for birds and humans using the DPD framework.

Key Results

  • Accuracy Gains: On the CUB200-2010 dataset, DPD achieved 34.5% mean accuracy, outperforming the previous state-of-the-art (Bubblebank) at 32.5%. On the 2011 dataset, it reached 50.98%, a significant jump over the 42.5% baseline.
  • Human Attributes: DPD outperformed Poselet-based methods on nearly all categories (e.g., 83.7% AP for "is male" vs. 82.4% for Poselets).
  • Efficiency: By moving away from Poselet ensembles to DPMs, the authors reduced processing time from ~10 seconds per image to several images per second, making the technique roughly 30 times faster than previous leading methods.
  • Ablation Study: The authors proved that localizing the head is the most critical factor for bird recognition; removing the head features caused accuracy to drop significantly (from ~50% to ~43%).

Conclusion and Future Work

DPD demonstrates that efficient pose-normalization is viable for real-world applications. However, the authors identify a few areas for improvement. The current model doesn't explicitly handle parts that are hidden behind other parts (self-occlusion). Additionally, the "weak" pooling method uses a relatively simple overlap calculation; more sophisticated optimization could potentially yield better correspondences. Finally, while this paper uses hand-crafted features, the authors acknowledge that deep convolutional features could likely be integrated into the DPD framework for even higher performance.

Summary updated

Abstract

Recognizing objects in fine-grained domains can be extremely challenging due to the subtle differences between subcategories. Discriminative markings are often highly localized, leading traditional object recognition approaches to struggle with the large pose variation often present in these domains. Pose-normalization seeks to align training exemplars, either piecewise by part or globally for the whole object, effectively factoring out differences in pose and in viewing angle. Prior approaches relied on computationally-expensive filter ensembles for part localization and required extensive supervision. This paper proposes two pose-normalized descriptors based on computationally-efficient deformable part models. The first leverages the semantics inherent in strongly-supervised DPM parts. The second exploits weak semantic annotations to learn cross-component correspondences, computing pose-normalized descriptors from the latent parts of a weakly-supervised DPM. These representations enable pooling across pose and viewpoint, in turn facilitating tasks such as fine-grained recognition and attribute prediction. Experiments conducted on the Caltech-UCSD Birds 200 dataset and Berkeley Human Attribute dataset demonstrate significant improvements of our approach over state-of-art algorithms.
BibTeX
@inproceedings{zhang2013deformable,
  author = {Ning Zhang and Ryan Farrell and Forrest N. Iandola and Trevor Darrell},
  title = {{Deformable Part Descriptors for Fine-Grained Recognition and Attribute Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=HyZJPWfdWB},
}
Figure