Appearance Based Qualitative Image Description for Object Class Recognition

Johan Thureson, Stefan CarlssonECCV 2004 · DOI

TL;DR

This paper introduces a novel image descriptor based on "order types" of gradient directions from triplets of points. This descriptor captures within-class appearance variations and achieves competitive results on object recognition tasks, outperforming previous methods on certain datasets, especially when dealing with background clutter.

What Problem Does This Paper Address?

Recognizing object classes (e.g., "car" vs. "motorcycle") is significantly harder than recognizing specific instances (e.g., "my car"). This is due to the inherent "within-class variation" – objects of the same class can look very different due to changes in viewpoint, illumination, and even subtle shape differences. Traditional methods often rely on comparing small image fragments, which can fail when variations are large or when objects are occluded. The challenge is to develop a method that can capture these variations effectively while still distinguishing between different classes, ideally by considering the whole object rather than just parts.

Methods

The core of this paper is a new image descriptor called the "order type" descriptor. Here's how it works:

  1. Gradient Detection: The image is first smoothed, and then gradient information (direction and strength) is extracted.
  2. Triplet Selection: The method considers random triplets of points within the image that have significant gradient information.
  3. Order Type Computation: For each triplet of points, the algorithm considers the directions of the gradients at those points, as well as the directions of the lines connecting pairs of points. By ranking these six directions and considering the polarity (direction) of the gradients, a unique "order type index" is generated. This index essentially captures the qualitative geometric relationship between the points and their local gradient structures.
  4. Histogram Descriptor: A histogram is built by counting the occurrences of these order type indices across all possible triplets in an image. The strength of the gradients is used to weight these occurrences. This histogram serves as the image descriptor.
  5. Similarity Measurement: Images are compared using a simple normalized inner product of their histogram descriptors.

The novelty lies in using these "order types" derived from triplets of points to create a holistic descriptor that is less sensitive to minor deformations and more robust to variations than methods relying on fixed-size fragments or solely edge-based approaches.

Datasets

The paper evaluates its method on a database containing four categories of images:

  • Faces
  • Airplanes
  • Motorcycles
  • Cars

This database also includes a set of background images. The authors mention that this is the same database used in reference [6], which is a common benchmark in the field.

Key Insights and Ideas

  • Qualitative Structure is Key: The paper argues that focusing on qualitative relationships between image features (like gradient directions) rather than precise pixel values or exact edge locations can be more robust to variations within an object class.
  • Triplets Capture Local Structure: Using triplets of points allows the method to capture more complex local image structures than pairwise comparisons, providing a richer description.
  • Holistic vs. Fragment-based: The proposed method aims to describe the entire image, overcoming the limitations of fragment-based approaches that struggle with large within-class variations or occlusions.
  • Bridging Appearance and Structure: The "order type" descriptor, based on gradient information, acts as a bridge between purely appearance-based methods and structure-based methods, leveraging both.

Key Results

  • Competitive Performance: The method achieves competitive results on object recognition tasks. For two categories (cars and motorcycles), it outperforms previous state-of-the-art results reported in [6].
  • Background Clutter Impact: For categories with significant background clutter (faces and airplanes), the whole-image approach initially yielded worse results than [6]. However, when images were manually cropped to focus on the object, performance significantly improved, demonstrating the method's potential when combined with a search mechanism.
  • Similarity Measure Effectiveness: Experiments with motorbikes, faces, and silhouette images of rabbits show that the inner product of the histogram descriptors effectively captures similarity within classes and discriminates between different classes.

Limitations and Future Work

  • Background Clutter: The primary limitation identified is the impact of background clutter on recognition accuracy, especially for categories where the object occupies a smaller portion of the image.
  • Search Algorithm Needed: For cluttered images, the authors suggest that combining their descriptor with an algorithm to find the optimal size and location of a window covering the object would significantly improve performance.
  • Fragment-based Descriptors: The authors acknowledge that fragment-based descriptors can be useful for recognizing occluded objects and for richer descriptions. They plan to explore such descriptors in the future, where the flexibility of their method's size handling will be important.
  • Tuning for Structure: The method can be tuned to pay more attention to strong structural properties, potentially forming a stronger bridge between appearance and structure-based recognition.

Summary updated

BibTeX
@inproceedings{thureson2004appearance,
  author = {Johan Thureson and Stefan Carlsson},
  title = {{Appearance Based Qualitative Image Description for Object Class Recognition}},
  booktitle = {European Conference on Computer Vision},
  year = {2004},
  doi = {10.1007/978-3-540-24671-8\_41},
}
Figure