Object Recognition Under Multifarious Conditions: A Reliability Analysis and a Feature Similarity-Based Performance Estimation

Dogancan Temel, Jin-Sol Lee, Ghassan AlRegib

Abstract

In this paper, we investigate the reliability of online recognition platforms, Amazon Rekognition and Microsoft Azure, with respect to changes in background, acquisition device, and object orientation. We focus on platforms that are commonly used by the public to better understand their real-world performances. To assess the variation in recognition performance, we perform a controlled experiment by changing the acquisition conditions one at a time. We use three smartphones, one DSLR, and one webcam to capture side views and overhead views of objects in a living room, an office, and photo studio setups. Moreover, we introduce a framework to estimate the recognition performance with respect to backgrounds and orientations. In this framework, we utilize both handcrafted features based on color, texture, and shape characteristics and data-driven features obtained from deep neural networks. Experimental results show that deep learning-based image representations can estimate the recognition performance variation with a Spearman's rank-order correlation of 0.94 under multifarious acquisition conditions.

Introduction: The Real-World Challenge

While computer vision has reached human-level performance on standard benchmarks, the "real world" is rarely a standard benchmark. Most existing research focuses on simulated distortions—like adding digital blur or noise to an image. However, the physical reality of taking a photo—the acquisition conditions—is often overlooked. For developers using off-the-shelf APIs like Amazon Rekognition or Microsoft Azure, understanding how environmental factors like messy backgrounds or budget hardware impact accuracy is critical.

Methodology and Dataset

The researchers conducted a controlled experiment using the CURE-OR (Challenging Unreal and Real Environments for Object Recognition) dataset. Unlike datasets that use post-processing to simulate challenges, CURE-OR captures 100 different objects under various real-world physical setups across 1 million images.

An ideal reference image featuring a white background and frontal orientation.

The study systematically varied three factors:

  1. Background: Ranging from plain white to complex 3D environments like living rooms and offices.
  2. Device: Five different cameras, including three smartphones, a DSLR (Nikon D80), and a webcam (Logitech C920).
  3. Orientation: Five angles, including front, side, back, and overhead views.

To estimate performance without knowing the "ground truth" for every image, the authors proposed a feature similarity framework. This framework compares "challenging" images to an "ideal" reference image (typically a front view on a white background taken with a DSLR).

Key Insights: Backgrounds and Orientations

The experiment revealed that backgrounds act as "feature thieves." Objects are recognized most accurately against white backgrounds; as environments become more complex, accuracy drops because background textures "mask" the object's features or resemble other objects.

Orientation proved to be an even greater hurdle. While frontal views are easiest for AI, overhead views are the most difficult due to self-occlusion, where the object's own shape hides its identifying features.

A challenging acquisition condition featuring an overhead view and a complex background.

Performance Estimation via Deep Learning

A significant finding of the study is that deep learning-based image representations (specifically VGG networks) are incredibly sensitive to the same conditions that confuse commercial APIs. By measuring the similarity between the features of a "challenging" photo and an "ideal" one, the researchers could predict accuracy drops with a Spearman’s rank-order correlation of 0.94 using VGG16 features.

While deep features outperformed handcrafted ones overall, handcrafted features showed specific strengths:

  • Color-based features were most effective at capturing background variations.
  • Edge-based features were superior at capturing orientation changes.

Conclusion and Future Directions

Both Amazon and Microsoft platforms showed similar performance trends, suggesting these challenges are fundamental to current computer vision architectures. The primary limitation of the current framework is its "full-reference" nature, requiring an ideal image for comparison. Future work aims to develop no-reference estimation algorithms that can assess reliability from a single, isolated image, potentially enabling real-time monitoring in deployed AI systems.

Summary updated

Abstract

In this paper, we investigate the reliability of online recognition platforms, Amazon Rekognition and Microsoft Azure, with respect to changes in background, acquisition device, and object orientation. We focus on platforms that are commonly used by the public to better understand their real-world performances. To assess the variation in recognition performance, we perform a controlled experiment by changing the acquisition conditions one at a time. We use three smartphones, one DSLR, and one webcam to capture side views and overhead views of objects in a living room, an office, and photo studio setups. Moreover, we introduce a framework to estimate the recognition performance with respect to backgrounds and orientations. In this framework, we utilize both handcrafted features based on color, texture, and shape characteristics and data-driven features obtained from deep neural networks. Experimental results show that deep learning-based image representations can estimate the recognition performance variation with a Spearman's rank-order correlation of 0.94 under multifarious acquisition conditions.
BibTeX
@inproceedings{temel2019object,
  author = {Dogancan Temel and Jin-Sol Lee and Ghassan AlRegib},
  title = {{Object Recognition Under Multifarious Conditions: A Reliability Analysis and a Feature Similarity-Based Performance Estimation}},
  booktitle = {ICIP},
  year = {2019},
  doi = {10.1109/icip.2019.8803317},
}
Figure