Salient Object Detection for Images Taken by People With Vision Impairments

Jarek Reynolds, Chandra Kanth Nagesh, Danna GurariWACV 2024

TL;DR

This paper introduces VizWiz-SalientObject, the first salient object detection dataset comprised of images taken by visually impaired individuals. They demonstrate that existing models struggle with this dataset due to its unique characteristics, like a high prevalence of text and large, less complex salient objects, highlighting a significant domain shift and the need for more robust models.

What Problem Does This Paper Address?

Salient object detection (SOD) aims to identify the most visually prominent object in an image. This technology has potential applications for people with vision impairments, such as helping them manage privacy by blurring backgrounds or focusing on specific content for descriptions. However, existing SOD datasets are typically built using high-quality images from the internet, which may not reflect the distinct characteristics of images captured by visually impaired individuals. These images can suffer from quality issues and often contain different types of content, particularly text. This paper addresses the gap by creating a new dataset and evaluating how well current SOD models perform on it, revealing their limitations in real-world scenarios faced by this community.

Methods

The core contribution is the creation of the VizWiz-SalientObject (VizWiz-SO) dataset. This involved:

  • Image Source: Leveraging images from the VizWiz-VQA-Grounding dataset, originally collected from visually impaired individuals seeking assistance with their photos.
  • Annotation Process: A rigorous crowdsourcing approach using Amazon Mechanical Turk (AMT) with strict quality control measures. Annotators were prompted to segment salient objects only after confirming the image wasn't a screenshot and contained a single, unambiguous foreground object.
  • Dataset Size: The final dataset contains 32,000 human-annotated images.
  • Benchmarking: Ten modern SOD models were evaluated on the VizWiz-SO dataset to assess their performance and identify weaknesses.

Datasets

  • VizWiz-SalientObject (VizWiz-SO): The newly introduced dataset, comprising 32,000 images taken by visually impaired individuals. It's characterized by a high prevalence of text in salient objects (68%), larger salient object coverage (~50%), and less complex boundaries compared to existing datasets.
  • Existing SOD Datasets: The paper compares VizWiz-SO to seven other SOD datasets: DUTS, DUT-OMRON, ECSSD, DAVIS-S, PASCAL-S, HRSOD, and UHRSD. These existing datasets are noted for being artificially constructed with high-quality internet images, often featuring centered salient objects with high contrast.

Key Insights and Ideas

  • Domain Shift is Real: The VizWiz-SO dataset represents a significant domain shift from traditional SOD datasets. The high percentage of text in salient objects (68% vs. a maximum of 24% in others) and the larger coverage area are key differentiators.
  • Text is a Crucial Cue: Models perform significantly worse on images lacking text in the salient object. This suggests that visual patterns associated with text are important cues for current SOD algorithms, and their absence poses a challenge.
  • Large Objects are Problematic: Current SOD models struggle with salient objects that occupy a large portion of the image. This is attributed to existing datasets often lacking such large objects, meaning models haven't been adequately trained to handle them.
  • Human Performance as a Benchmark: The paper establishes human performance as an upper bound, showing that even the best-performing model (InSPyReNet) nearly achieves it, but most other models fall considerably short.
  • Real-World Challenges Matter: Models designed for speed (e.g., BASNet) or compactness (e.g., U2Net) were among the poorer performers, highlighting the need for models that are both accurate and practical for real-world applications.

Key Results

  • Dataset Size: VizWiz-SO is the largest among the compared datasets with 32,000 human-annotated images.
  • Text Prevalence: 68% of salient objects in VizWiz-SO contain text, a stark contrast to other datasets.
  • Coverage: Salient objects in VizWiz-SO occupy, on average, ~50% of the image.
  • Model Performance: The top-performing model, InSPyReNet, achieved near-human performance on VizWiz-SO. However, most other models struggled, with MAE performance dropping significantly compared to their performance on other datasets. For instance, PGNet's MAE on VizWiz-SO was 0.2123, compared to 0.028 on DUTS-TE.
  • Fine-grained Analysis: Models performed worse on images with:
  • No text in the salient object.
  • Large coverage ratios (salient objects occupying more of the image).
  • Less complex boundaries.
  • Poorer image quality.

Limitations and Future Work

  • Limitations:
    • The paper notes that while InSPyReNet performs well, training on VizWiz-SO itself did not significantly boost performance for this model, suggesting that the model's architecture and general training might be robust enough, but the domain shift still presents challenges.
  • The VizWiz-VQA-Grounding dataset's annotations did not account for holes in salient objects, leading to lower IoU scores in some cases when comparing salient objects to answer groundings.
  • Future Work:
    • The authors suggest that the VizWiz-SO dataset will support the development of more generalized SOD models that can handle real-world challenges like lower image quality, large objects, and less complex boundaries.
  • They anticipate algorithmic improvements will benefit downstream applications such as salient object tracking, visual question answering, and image captioning.
  • The dataset is publicly shared to facilitate future research.

Summary updated

BibTeX
@inproceedings{reynolds2024salient,
  author = {Jarek Reynolds and Chandra Kanth Nagesh and Danna Gurari},
  title = {{Salient Object Detection for Images Taken by People With Vision Impairments}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure