Training Indoor and Scene-Specific Semantic Segmentation Models to Assist Blind and Low Vision Users in Activities of Daily Living

Ruijie Sun, Giles Hamilton-Fletcher, Sahil Faizal, Chen Feng, Todd E. Hudson, John-Ross Rizzo, Kevin C. ChanIEEE Open Journal of Engineering in Medicine and Biology 2025 · DOI

TL;DR

This paper demonstrates that training AI models for semantic segmentation on specific indoor environments (like kitchens or bedrooms) significantly improves their ability to identify objects crucial for daily living tasks for visually impaired individuals, outperforming generic indoor models. These specialized models are also more robust in low-light conditions, making them more practical for real-world assistive technologies.

What Problem Does This Paper Address?

Persons with blindness or low vision (pBLV) face significant challenges in performing everyday activities of daily living (ADLs) and instrumental activities of daily living (IADLs) independently. While AI-powered assistive technologies, particularly those using semantic segmentation, hold great promise for identifying objects and understanding environments, their effectiveness in diverse indoor settings and varying lighting conditions remains a critical question. Existing generic models, trained on a broad range of indoor scenes, may not be sufficiently precise for the specific needs of pBLV, leading to limitations in functionality and practicality. This research aims to bridge this gap by investigating whether scene-specific semantic segmentation models can offer superior performance for assisting pBLV in their daily routines.

Methods

The core of this research involves training and evaluating semantic segmentation models using the DeepLabV3+ architecture, a popular choice for smartphone-compatible computer vision tasks. The authors developed four scene-specific models, each trained on data from a particular indoor environment: kitchen, bedroom, bathroom, and living room. These were then compared against a single "generic" indoor model trained on a broader collection of indoor scenes.

The evaluation focused on two key aspects:

  1. Overall Segmentation Performance: Measured using mean accuracy, mean Intersection over Union (mIoU), and weighted Intersection over Union (wIoU). wIoU is particularly interesting as it accounts for the visual prevalence of objects.
  2. Key Object Identification: The models were assessed on their ability to segment the "top 5" most important objects for ADLs/IADLs within each room (e.g., dishwasher in the kitchen, bed in the bedroom).
  3. Low-Light Resilience: The models' performance was tested under artificially reduced lighting conditions to simulate real-world scenarios where lighting is not always optimal.

Datasets

The study utilized the MIT ADE20K SceneParse150 dataset. This dataset is described as a large, open-source collection of annotated indoor and outdoor scenes. For this research, the authors filtered it to include only indoor scenes and further curated specific training and validation datasets for the four target rooms (kitchen, bedroom, bathroom, living room), excluding irrelevant outdoor scenes.

Key Insights and Ideas

  • Specialization Trumps Generalization: The most striking finding is that models trained for specific rooms consistently outperform a generic indoor model. This suggests that tailoring AI models to the unique visual characteristics of different environments is crucial for effective assistive technology.
  • Room Uniqueness Matters: The degree of improvement with scene-specific models was more pronounced in rooms with more unique objects (e.g., bathrooms with bathtubs, bedrooms with beds). This highlights that the distinctiveness of objects within a scene is a key factor in the benefit of specialization.
  • Low-Light Advantage: Scene-specific models demonstrated greater resilience to low-light conditions. This is a critical insight, as lighting is often unpredictable and can significantly degrade the performance of computer vision systems, directly impacting the reliability of assistive tools for pBLV.
  • Key Object Focus: The scene-specific models showed a substantial advantage in identifying objects directly relevant to ADLs/IADLs, which is the primary goal of such assistive technologies.

Key Results

  • Overall Performance: Scene-specific models achieved higher mean accuracy and mIoU in the kitchen, bedroom, and bathroom compared to the generic model. For wIoU, scene-specific models outperformed the generic model across all four rooms.
  • Key Object Segmentation: Scene-specific models had higher mIoU for 19 out of 20 key object classes and higher mean accuracy for all 20 key object classes compared to generic models. For example, the mIoU for "bed" in bedrooms was significantly higher for the scene-specific model (0.6841) than the generic model (0.0504).
  • Low-Light Performance: While all models saw a decrease in performance with reduced brightness, scene-specific models generally showed smaller performance drops, indicating better robustness.

Limitations and Future Work

  • Living Room Anomaly: The generic model performed better than the scene-specific model in the living room. The authors attribute this to living rooms having more "room-agnostic" objects (like chairs and tables) that are common across many indoor spaces, where a broader model might be more effective.
  • Image Scale: The current study assessed room-level overviews. Future work should evaluate segmentation performance on closer views of objects as a user approaches them, ensuring robustness in active use.
  • Dynamic Model Switching: The authors suggest that a dynamic, context-aware system that switches between the best-performing scene-specific models in real-time would be the most effective approach for assistive devices. This could be triggered manually or automatically based on scene identification.
  • Further Enhancements: Future research could explore expanding data augmentation, integrating more sensor inputs, and refining image processing pipelines to further improve performance, especially in challenging lighting conditions.

Summary updated

Abstract

<i>Goal:</i> Persons with blindness or low vision (pBLV) face challenges in completing activities of daily living (ADLs/IADLs). Semantic segmentation techniques on smartphones, like DeepLabV3+, can quickly assist in identifying key objects, but their performance across different indoor settings and lighting conditions remains unclear. <i>Methods:</i> Using the MIT ADE20K SceneParse150 dataset, we trained and evaluated AI models for specific indoor scenes (kitchen, bedroom, bathroom, living room) and compared them with a generic indoor model. Performance was assessed using mean accuracy and intersection-over-union metrics. <i>Results:</i> Scene-specific models outperformed the generic model, particularly in identifying ADL/IADL objects. Models focusing on rooms with more unique objects showed the greatest improvements (bedroom, bathroom). Scene-specific models were also more resilient to low-light conditions. <i>Conclusions:</i> These findings highlight how using scene-specific models can boost key performance indicators for assisting pBLV across different functional environments. We suggest that a dynamic selection of the best-performing models on mobile technologies may better facilitate ADLs/IADLs for pBLV.
BibTeX
@article{sun2025training,
  author = {Ruijie Sun and Giles Hamilton-Fletcher and Sahil Faizal and Chen Feng and Todd E. Hudson and John-Ross Rizzo and Kevin C. Chan},
  title = {{Training Indoor and Scene-Specific Semantic Segmentation Models to Assist Blind and Low Vision Users in Activities of Daily Living}},
  journal = {IEEE Open Journal of Engineering in Medicine and Biology},
  year = {2025},
  doi = {10.1109/ojemb.2025.3607816},
}
Figure