Introduction
Machine learning models are fundamentally shaped by their training data. However, visual datasets are not neutral reflections of reality; they are "slices" of the world that often carry the historical and cultural biases of their creators and annotators. With modern datasets like COCO or OpenImages containing millions of images, manual auditing is impossible. Biases—such as geographic under-representation or stereotypical portrayals of gender—often go unnoticed until a model fails in a real-world application.
REVISE (REvealing VIsual biaSEs) addresses this gap by providing a programmatic, automated way to "interrogate" a dataset's annotations and visual content to find these anomalies before models are deployed.
Methodology
REVISE is implemented as a Jupyter notebook that utilizes statistical analysis and pre-trained computer vision models to audit datasets across three key modules:
- Object-Based Analysis: Calculates statistics on object frequency, scale (size relative to the image), and co-occurrence. It uses ResNet architectures to classify scenes and extract features for measuring "appearance diversity."
- Gender-Based Analysis: Uses a novel "scaled distance" metric to estimate the 3D distance between a person and an object, distinguishing between simple proximity and actual interaction.
- Geography-Based Analysis: Utilizes geolocation metadata and language detection (via fastText) to identify whether a region is portrayed through a local perspective or a "tourist gaze."
Dataset Auditing
The authors validated REVISE by analyzing three influential datasets: COCO (Common Objects in Context), OpenImages, and YFCC100m. These datasets provide the complex annotations—such as bounding boxes and segmentations—that REVISE interrogates to find hidden patterns.
Key Findings
The "Male-as-Norm" Default
One of the most striking findings was that when a person in an image is too small to be clearly identified, annotators default to labeling them as "male" 69% to 77% of the time. This bias is especially prevalent in "outdoor sports" scenes.
Interaction vs. Presence
The tool revealed subtle biases in how genders interact with objects. For example, in images containing the "organ" (musical instrument) tag, males were statistically more likely to be positioned in a way that suggested they were actually playing the instrument, while females were often just standing near it.
Scale and Geographic Stereotyping
REVISE found that 77% of airplanes in COCO are "extra large," occupying more than 9.3% of the image, because photos are typically taken on the ground rather than in the sky. Geographically, the tool identified that the USA is drastically overrepresented in YFCC100m, while the entire continent of Africa is sparsely represented. Furthermore, in developing nations, a high percentage of images are tagged in non-local languages, leading to stereotypical representations (e.g., Kiribati being represented almost exclusively by "wildlife" tags).
Mitigation and Impact
Unlike previous bias detection research, REVISE provides actionable mitigation strategies. If the tool finds that "person" co-occurs more with "unhealthy food" than "vegetables," it explicitly suggests querying for images of people with healthier food options to balance the dataset. Because REVISE looks at the dataset and its annotations directly (rather than just model outputs), it can find biases that are "baked in" before a single weight is trained, making it a powerful tool for model agnosticism.
Limitations and Future Work
REVISE relies on pre-trained models that may themselves contain biases. Additionally, current gender analysis is limited to a binary classification because those are the labels available in major datasets. The authors emphasize that while REVISE surfaces patterns, the final judgment of whether a pattern is "problematic" requires a human user to consider historical and cultural contexts. Future work aims to expand the tool to include other axes of identity, such as race and age.
REVISE is open-source and available at: https://github.com/princetonvisualai/revise-tool
Summary updated