Indoor image representation by high-level semantic features

Chiranjibi Sitaula, Yong Xiang, Yushu Zhang, Xuequan Lu, Sunil Aryal

Abstract

Indoor image classification is a complex task due to the high degree of inter-class similarity and intra-class dissimilarity. This paper introduces a novel approach that utilizes high-level semantic features based on object co-occurrence patterns. By constructing a semantic object dictionary and employing a weighting mechanism termed "Delta Parameters," the proposed method captures the contextual relationships between objects. Evaluated on major benchmarks including MIT-67, Scene15, and NYU V1, the approach demonstrates state-of-the-art accuracy while maintaining a compact feature representation.

Introduction

Unlike outdoor scenes, which are often characterized by global textures like sky or water, indoor environments are primarily defined by the objects they contain. This leads to two significant challenges in computer vision: inter-class similarity, where different room types (e.g., a kitchen and a library) share common objects like tables and chairs; and intra-class dissimilarity, where the same category of room can vary wildly in architectural style and lighting.

Challenges in indoor classification: Two distinct library environments demonstrating intra-class dissimilarity.

Traditional low-level and mid-level features often fail to capture the necessary context to resolve these ambiguities. This research proposes moving beyond simple object detection toward "semantic object association," treating the relationships between objects as the primary descriptor for scene identity.

Methodology

The proposed system extracts high-level semantic features through a four-stage pipeline:

  1. Object Pattern Dictionary Construction: Images are partitioned into sub-image slices. A pre-trained Inception V3 model identifies the top 10 objects within each slice. A dictionary is then built to map these objects and their co-occurrence frequencies, a technique inspired by n-gram models in Natural Language Processing (NLP).
The proposed pipeline for extracting raw objects from image slices using Inception V3.
  1. Semantic Object Extraction: Raw object detections are filtered into "semantic" objects using logical propositions. Objects that consistently appear together across multiple slices are grouped into single semantic units that define the scene's context.
  2. Probability and Delta Parameter Calculation: To weigh the importance of specific objects, the authors introduce "Delta Parameters." For example, while both chairs and books appear in libraries, a book is a much stronger indicator of the category. Six variations of these parameters were tested to optimize classification.
  3. Feature Fusion and Classification: Final features are calculated as the product of an object's category probability and its Delta Parameter. These high-level features are then classified using a Support Vector Machine (SVM).

Experimental Results

The methodology was validated against three benchmark datasets: MIT-67 (15,620 images), Scene15 (4,485 images), and NYU V1 (2,284 images). The results indicate a significant improvement over previous state-of-the-art (SOTA) methods:

  • MIT-67: Achieved 94.1% accuracy, surpassing the previous SOTA of 86.7%.
  • Scene15: Reached 98.9% accuracy.
  • NYU V1: Reached 96.5% accuracy.

The "Normal Delta Parameter" emerged as the most effective weighting mechanism for distinguishing between highly similar classes.

Conclusion and Future Work

This research demonstrates that high-level semantic associations provide a more robust representation for indoor scenes than traditional features. By treating visual scenes through a linguistic lens (object co-occurrence as n-grams), the model achieves superior accuracy with lower-dimensional features. Future work will explore more complex object associations beyond 2-gram pairs and investigate the integration of diverse deep learning backbones to further enhance the semantic dictionary's quality.

Summary updated

BibTeX
@article{sitaula2019indoor,
  author = {Chiranjibi Sitaula and Yong Xiang and Yushu Zhang and Xuequan Lu and Sunil Aryal},
  title = {{Indoor image representation by high-level semantic features}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2925002},
}
Figure