Indoor Image Representation by High-Level Semantic Features

Chiranjibi Sitaula, Yong Xiang, Yushu Zhang, Xuequan Lu, Sunil Aryal

TL;DR

This paper introduces a novel method for indoor scene classification that moves beyond identifying individual objects to understanding their semantic associations. By building a domain-specific "Object Pattern Dictionary" inspired by natural language processing, the researchers extract high-level semantic features that achieve state-of-the-art accuracy on major benchmarks while maintaining a significantly lower feature dimensionality than previous methods.

What Problem Does This Paper Address?

Indoor image classification is notoriously difficult for computer vision systems. Unlike outdoor scenes (like a beach or a forest) which often have distinct global textures and colors, indoor scenes are defined by the objects within them. However, two major hurdles persist:

  1. Intra-class Dissimilarity: Two libraries can look completely different in terms of layout, lighting, and architecture (see Figure 1).
  2. Inter-class Similarity: A kitchen and a library might both contain tables and chairs. If a model only looks for the presence of a "table," it gets "confused."

Existing low-level (pixel/color) and middle-level (shape/part) features often fail to capture the context that distinguishes these scenes. While some high-level features exist, they are often extremely high-dimensional, creating a significant computational burden.

![Fig.

  1. Two library images that look dissimilar.](/figures/art_69f91ff7f7fb423bbe62b0a115c5cedd)

Methods

The authors propose a four-step pipeline to extract high-level semantic features:

  1. Image Slicing: To focus on local object information rather than just global layout, the image is sliced into sub-images (e.g., a 3x3 or 4x4 grid).
  2. Raw Object Extraction: Each slice is fed into a pre-trained Inception V3 model to identify the top 10 most probable objects.
  3. Object Pattern Dictionary Construction: Inspired by the n-gram model in Natural Language Processing (NLP), the authors build a dictionary that records how often pairs of objects co-occur within specific scene categories. For example, "microwave" and "bread" have a high co-occurrence in a kitchen.
  4. Semantic Feature Extraction: The system retrieves "semantic objects" from the dictionary based on the raw objects found in the image. It then calculates a final feature vector by fusing the probability of these objects with a delta parameter—a weighting factor designed to highlight objects that are most discriminative for a particular category.

![Fig.

  1. Detailed diagram for the extraction of high-level semantic features.](/figures/art_7f5947a171674e3bb8d055c9b6f0b6e8)

Datasets

The researchers evaluated their approach on three widely recognized public datasets:

  • MIT-67: The largest dataset used, containing 15,620 images across 67 indoor categories.
  • Scene15: A mix of 15 indoor and outdoor categories (4,485 images).
  • NYU V1: A smaller dataset focused on 7 indoor categories (2,284 images).

Key Insights and Ideas

  • Context over Content: The core "aha moment" is that the relationship between objects (co-occurrence) is more informative for scene recognition than the mere presence of the objects themselves.
  • NLP for Vision: Borrowing the concept of n-grams to model object associations allows the system to handle the "semantic noise" of common objects (like chairs) appearing in multiple different types of rooms.
  • Dimensionality Efficiency: By focusing on a refined set of semantic associations rather than raw deep learning activations, the authors produced features that are more compact yet more descriptive, leading to better performance with less data "weight."

Key Results

The proposed method significantly outperformed existing state-of-the-art (SOTA) techniques across all tested datasets:

  • MIT-67: Achieved 94.1% accuracy, a substantial jump from the previous SOTA "Objectness" method (86.7%) and nearly triple the accuracy of traditional low-level features like GIST (26.1%).
  • Scene15: Reached 98.9% accuracy, outperforming deep learning models like DUCA (94.5%).
  • NYU V1: Achieved 96.5% accuracy, compared to 80.6% for DUCA.

The ablation studies also revealed that a dictionary size of roughly 16,000 objects (derived from 4x4 image slicing) provided the best balance for classification accuracy.

Limitations and Future Work

While highly effective, the authors note a few areas for improvement:

  • Complexity: The extraction of semantic objects is the most computationally expensive part of the pipeline, with a worst-case time complexity of $O(m^3)$ relative to the number of categories.
  • Static Associations: The current model relies on pre-defined propositions for object association. Future work could explore more complex, non-linear associations or end-to-end learning where the dictionary and features are optimized simultaneously.
  • Dependency: The system's performance is inherently tied to the accuracy of the underlying object detection model (Inception V3). Errors in initial object tagging will propagate through the semantic mapping.

Summary updated

Abstract

Indoor image features extraction is a fundamental problem in multiple fields such as image processing, pattern recognition, robotics and so on. Nevertheless, most of the existing feature extraction methods, which extract features based on pixels, color, shape/object parts or objects on images, suffer from limited capabilities in describing semantic information (e.g., object association). These techniques, therefore, involve undesired classification performance. To tackle this issue, we propose the notion of high-level semantic features and design four steps to extract them. Specifically, we first construct the objects pattern dictionary through extracting raw objects in the images, and then retrieve and extract semantic objects from the objects pattern dictionary. We finally extract our high-level semantic features based on the calculated probability and delta parameter. Experiments on three publicly available datasets (MIT-67, Scene15 and NYU V1) show that our feature extraction approach outperforms state-of-the-art feature extraction methods for indoor image classification, given a lower dimension of our features than those methods.
BibTeX
@article{sitaula2019indoor,
  author = {Chiranjibi Sitaula and Yong Xiang and Yushu Zhang and Xuequan Lu and Sunil Aryal},
  title = {{Indoor Image Representation by High-Level Semantic Features}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2925002},
}

Loading related content...

Figure