REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets

Angelina Wang, Arvind Narayanan, Olga Russakovsky

Introduction

Machine learning models are fundamentally shaped by their training data. However, visual datasets are not neutral reflections of reality; they are "slices" of the world that often carry the historical and cultural biases of their creators and annotators. With modern datasets like COCO or OpenImages containing millions of images, manual auditing is impossible. Biases—such as geographic under-representation or stereotypical portrayals of gender—often go unnoticed until a model fails in a real-world application.

REVISE (REvealing VIsual biaSEs) addresses this gap by providing a programmatic, automated way to "interrogate" a dataset's annotations and visual content to find these anomalies before models are deployed.

Methodology

REVISE is implemented as a Jupyter notebook that utilizes statistical analysis and pre-trained computer vision models to audit datasets across three key modules:

  • Object-Based Analysis: Calculates statistics on object frequency, scale (size relative to the image), and co-occurrence. It uses ResNet architectures to classify scenes and extract features for measuring "appearance diversity."
  • Gender-Based Analysis: Uses a novel "scaled distance" metric to estimate the 3D distance between a person and an object, distinguishing between simple proximity and actual interaction.
  • Geography-Based Analysis: Utilizes geolocation metadata and language detection (via fastText) to identify whether a region is portrayed through a local perspective or a "tourist gaze."

Dataset Auditing

The authors validated REVISE by analyzing three influential datasets: COCO (Common Objects in Context), OpenImages, and YFCC100m. These datasets provide the complex annotations—such as bounding boxes and segmentations—that REVISE interrogates to find hidden patterns.

REVISE analyzes datasets like COCO and OpenImages, which feature complex object detection, segmentation, and contextual annotations.

Key Findings

The "Male-as-Norm" Default

One of the most striking findings was that when a person in an image is too small to be clearly identified, annotators default to labeling them as "male" 69% to 77% of the time. This bias is especially prevalent in "outdoor sports" scenes.

Interaction vs. Presence

The tool revealed subtle biases in how genders interact with objects. For example, in images containing the "organ" (musical instrument) tag, males were statistically more likely to be positioned in a way that suggested they were actually playing the instrument, while females were often just standing near it.

Scale and Geographic Stereotyping

REVISE found that 77% of airplanes in COCO are "extra large," occupying more than 9.3% of the image, because photos are typically taken on the ground rather than in the sky. Geographically, the tool identified that the USA is drastically overrepresented in YFCC100m, while the entire continent of Africa is sparsely represented. Furthermore, in developing nations, a high percentage of images are tagged in non-local languages, leading to stereotypical representations (e.g., Kiribati being represented almost exclusively by "wildlife" tags).

Geographic analysis in REVISE highlights the stark imbalance in image distribution and frequency across different global regions.

Mitigation and Impact

Unlike previous bias detection research, REVISE provides actionable mitigation strategies. If the tool finds that "person" co-occurs more with "unhealthy food" than "vegetables," it explicitly suggests querying for images of people with healthier food options to balance the dataset. Because REVISE looks at the dataset and its annotations directly (rather than just model outputs), it can find biases that are "baked in" before a single weight is trained, making it a powerful tool for model agnosticism.

Limitations and Future Work

REVISE relies on pre-trained models that may themselves contain biases. Additionally, current gender analysis is limited to a binary classification because those are the labels available in major datasets. The authors emphasize that while REVISE surfaces patterns, the final judgment of whether a pattern is "problematic" requires a human user to consider historical and cultural contexts. Future work aims to expand the tool to include other axes of identity, such as race and age.

REVISE is open-source and available at: https://github.com/princetonvisualai/revise-tool

Summary updated

Abstract

Machine learning models are known to perpetuate and even amplify the biases present in the data. However, these data biases frequently do not become apparent until after the models are deployed. To tackle this issue and to enable the preemptive analysis of large-scale dataset, we present our tool. REVISE (REvealing VIsual biaSEs) is a tool that assists in the investigation of a visual dataset, surfacing potential biases currently along three dimensions: (1) object-based, (2) gender-based, and (3) geography-based. Object-based biases relate to size, context, or diversity of object representation. Gender-based metrics aim to reveal the stereotypical portrayal of people of different genders. Geography-based analyses consider the representation of different geographic locations. REVISE sheds light on the dataset al.ong these dimensions; the responsibility then lies with the user to consider the cultural and historical context, and to determine which of the revealed biases may be problematic. The tool then further assists the user by suggesting actionable steps that may be taken to mitigate the revealed biases. Overall, the key aim of our work is to tackle the machine learning bias problem early in the pipeline. REVISE is available at https://github.com/princetonvisualai/revise-tool .
BibTeX
@inproceedings{wang2020revise,
  author = {Angelina Wang and Arvind Narayanan and Olga Russakovsky},
  title = {{REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58580-8\_43},
}
Figure