Abstract
The paper analyzes the accuracy of publicly available object-recognition systems on a geographically diverse dataset. This dataset contains household items and was designed to have a more representative geographical coverage than commonly used image datasets in object recognition. We find that the systems perform relatively poorly on household items that commonly occur in countries with a low household income. Qualitative analyses suggest the drop in performance is primarily due to appearance differences within an object class (e.g., dish soap) and due to items appearing in a different context (e.g., toothbrushes appearing outside of bathrooms). The results of our study suggest that further work is needed to make object-recognition systems work equally well for people across different countries and income levels.
Introduction
Current object-recognition systems exhibit a significant "wealth bias," performing substantially better on household items from high-income, Western countries than on those from lower-income or non-Western regions. While object recognition has reached impressive milestones in benchmarks like ImageNet, these systems are increasingly deployed in high-stakes real-world applications—from assistive technology for the visually impaired to global image search. However, most training data is harvested from the internet in a way that heavily over-represents the United States and Europe, creating systemic "blind spots" in our most advanced visual AI.
Methodology and Datasets
To investigate this bias, researchers performed a large-scale audit of six existing systems: five commercial APIs (Microsoft Azure, Clarifai, Google Cloud Vision, Amazon Rekognition, and IBM Watson) and one state-of-the-art research model.
The study utilized the Dollar Street dataset, which captures everyday life across different income levels. It contains photos of 135 classes of household items taken in 264 homes across 54 countries, with each photo tagged with the household's monthly income. This stands in stark contrast to standard computer vision benchmarks like ImageNet or COCO, which are heavily skewed toward Western regions.
Key Insights: Appearance, Context, and Language
The performance drop observed in lower-income regions is driven by three primary factors:
- Appearance Differences: Items like stoves or soap look drastically different across socio-economic strata (e.g., a wood-fired stove vs. an electric range).
- Contextual Shifts: Models often rely on surroundings to identify objects. A toothbrush in a high-income home is almost always in a bathroom; in a low-income home, it might be kept in a kitchen or outdoors, confusing models trained on Western domestic norms.
- Linguistic Bias: Most datasets are built using English keywords. This ignores cultural variations that don't translate directly, effectively baking Western visual norms into the model's foundation.
Results: The Income and Geography Gap
The audit revealed a clear correlation: as household income increased, so did the model's accuracy. Models were approximately 10% less accurate for households earning less than $50/month compared to those earning over $3,500/month. Furthermore, there was a 15–20% absolute difference in accuracy when recognizing items in the US compared to countries like Somalia or Burkina Faso.
Qualitative failures were striking; for example, a bottle of dish soap from a lower-income household in Nepal was frequently misidentified as "food," "cheese," or "bread" by major APIs, while Western equivalents were identified with high confidence.
Conclusion and Future Directions
This research identifies a systemic issue in how computer vision models are trained and evaluated. Simply "adding more data" is insufficient if that data maintains existing biases. The authors suggest that future work must focus on geographic resampling, multi-lingual training data collection, and developing algorithmic robustness to ensure that object recognition works equitably for everyone, regardless of their geography or income level.
Summary updated