Context-based indoor object detection as an aid to blind persons accessing unfamiliar environments

Xiaodong Yang, Xiaodong Yang 0001, Yingli Tian, Chucai Yi, Aries ArditiACM Multimedia 2010 · DOI

TL;DR

This paper introduces a computer vision system to help blind individuals navigate unfamiliar indoor environments by detecting doors and identifying their purpose (e.g., office, lab, bathroom) using text recognition from signs. By combining robust door detection with OCR on associated signage, the system aims to provide crucial contextual information for independent wayfinding.

What Problem Does This Paper Address?

Independent travel for blind and visually impaired individuals presents significant challenges, especially in unfamiliar indoor spaces. While GPS is useful outdoors, indoor navigation aids are lacking. Existing systems often rely on complex, expensive multi-sensor setups. This paper tackles the problem of enabling blind individuals to independently identify and understand their surroundings within buildings, specifically by locating and classifying doors and other important amenities like elevators. The core challenge lies in the inherent similarity of indoor objects (e.g., doors, elevators, cabinets) and the variability in their appearance across different environments, making simple shape-based detection insufficient.

Methods

The proposed system integrates two main components: door detection and text-based object classification.

  • Door Detection: The authors build upon a previously developed algorithm that uses general geometric shape, combining edges and corners to detect doors and elevators. This method is designed to be robust to variations in color, texture, and occlusion, and aims to handle both intra-class variations (different types of doors) and small inter-class differences (doors vs. elevators). They also introduce a method to distinguish concave objects (like elevator doors recessed into walls) from convex objects (like bookshelves protruding from walls) by analyzing the relative positions of detected "doorframe" features and their lateral components.
  • Text Extraction and Recognition: To differentiate between similar objects (e.g., an office door vs. a bathroom door), the system extracts and recognizes text from associated signage.
  • Color Decomposition: The process begins by decomposing images into layers based on dominant colors, assuming text strings generally have uniform colors against a white background.
  • Text Region Extraction: A sliding window approach is used to identify regions with high binary switch frequencies, indicative of character strokes. Accumulated horizontal and vertical projections then refine these regions.
  • Text Localization: A text structure model and layout analysis (considering area variance and character spacing) are employed to filter out non-text regions and localize individual characters.
  • Text Recognition: Off-the-shelf Optical Character Recognition (OCR) software (Tesseract and OmniPage) is used to convert the extracted text into readable codes.

The final output is the object type, orientation, and location, which can be presented to the user as speech.

Datasets

The paper describes constructing a database of 221 images collected from various environments for testing. This database includes both door and non-door images (like bookshelves and cabinets) and is categorized into "Simple," "Medium," and "Complex" based on background complexity, deformation, occlusion, illumination, and scale variations. The paper does not mention using pre-existing public datasets for training or evaluation, implying this is a custom dataset.

Key Insights and Ideas

  • Context is King: Simple geometric shape detection is insufficient for distinguishing functionally different indoor objects. Leveraging contextual information, specifically text on signage, is crucial for accurate classification.
  • Concave vs. Convex for Doors: The distinction between recessed (concave) and protruding (convex) door-like structures is a clever geometric cue that helps differentiate functional doors/elevators from decorative or storage elements like bookshelves.
  • Robust Text Extraction for Indoor Signs: Indoor signage often has complex backgrounds and varied colors. The proposed color decomposition and binary switch analysis offer a robust way to extract text regions even in challenging conditions.
  • OCR as a Final Step: The paper highlights that raw images are not suitable for OCR; effective pre-processing for text region extraction and localization is a necessary precursor for accurate text recognition.

Key Results

The proposed door detection algorithm, when combined with convexity detection, achieved an 89.5% true positive rate with a 2.3% false positive rate on average across their categorized dataset. Without convexity detection, the average rates were 92.3% true positive and 5.0% false positive. The paper notes that convexity detection is particularly effective in reducing false positives in more complex environments, which is desirable for user safety. The text extraction and localization algorithm is described as performing well, enabling OCR to produce readable codes from extracted text regions.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that in highly complex backgrounds, some background elements can still interfere with text extraction, and pixel-based algorithms struggle to distinguish them from actual text.
  • The performance of door detection decreases with increasing complexity of the environment (lower true positive rate and higher false positive rate in "Complex" categories).
  • The system relies on the presence and readability of text on signage.
  • Future Work:
    • Detecting and recognizing more types of indoor objects and icons beyond doors and text.
  • Investigating significant human interface issues, including auditory output and spatial updating of object location, orientation, and distance.
  • Developing real-time updates to enhance the user's ability to build spatial memory of their surroundings.

Summary updated

BibTeX
@inproceedings{yang2010context,
  author = {Xiaodong Yang and Xiaodong Yang and Yingli Tian and Chucai Yi and Aries Arditi},
  title = {{Context-based indoor object detection as an aid to blind persons accessing unfamiliar environments}},
  booktitle = {ACM Multimedia},
  year = {2010},
  doi = {10.1145/1873951.1874156},
}
Figure