Object-to-Scene: Learning to Transfer Object Knowledge to Indoor Scene Recognition

Bo Miao, Liguang Zhou, Ajmal Saeed Mian, Tin Lun Lam, Yangsheng Xu

Introduction: The Challenge of Indoor Granularity

Indoor scene recognition is a notoriously difficult task for computer vision. Unlike outdoor scenes with distinct horizons or natural landmarks, indoor environments like "home offices" and "dining rooms" often share global features such as similar walls, floors, and lighting.

Standard Convolutional Neural Networks (CNNs) frequently fail because they lack focus on the "granularity" of objects. A model might identify a large table and chairs and predict "dining room," while overlooking a small stove or microwave that identifies the space as a "kitchen." The authors of this paper argue that an intelligent system must understand not just what objects are present, but the contextual relationships (co-occurrence) between them—the "semantic glue" of a room.

Comparison of ResNet50 and OTS performance on a kitchen scene, highlighting the impact of object segmentation.

Methodology: The OTS Framework

The Object-to-Scene (OTS) framework is a one-stream architecture designed to capture object-level features and their global relations through three primary modules.

The OTS framework architecture, illustrating the flow from segmentation to global relation aggregation.

1. Object Feature Aggregation Module (OFAM)

Rather than learning from scratch, OTS leverages a pre-trained segmentation network (e.g., PSPNet) to identify objects within a scene. The OFAM takes feature maps from the backbone and applies masks based on the segmentation results. This process generates a dedicated feature vector for every detected object category (e.g., "chair," "stove," "lamp").

2. Object Attention Module (OAM)

The OAM serves as the system's reasoning engine. It employs a cascaded attention mechanism to learn how objects relate to one another. For instance, it learns that the co-occurrence of a "bed" and a "lamp" strongly indicates a bedroom. To maintain efficiency, the OAM uses a compression factor to reduce channel dimensions, making it faster and lighter than standard self-attention mechanisms.

3. Global Relation Aggregation Module (GRAM)

To produce a final scene label, the system must fuse individual object relations. The GRAM utilizes "strip depthwise convolution," which operates across the list of object features rather than spatial pixels. This allows the model to synthesize the relational data into a single, robust scene representation vector.

Key Insights: Objects as Scene Vocabulary

The core philosophy of OTS is that a scene is the sum of its objects and their interactions. By forcing the network to analyze object-level features, the model becomes more robust to cluttered or unconventional room layouts.

The attention module allows the model to implicitly learn a probability distribution of objects, "attending" to highly discriminative items (like a stove in a kitchen) while down-weighting generic elements like walls or floors. Furthermore, the authors found that concatenating features within the attention block was more effective than summing them, as it preserves object identity alongside relational context.

Experimental Results

The researchers evaluated OTS on the Places365 and SUN-RGBD datasets.

  • Superior Accuracy: On the Places365-7classes dataset, OTS achieved 90.1% accuracy, significantly outperforming the ResNet50 baseline (82.7%) and previous state-of-the-art methods (88.1%).
  • Generalization: When tested on the SUN-RGBD dataset without prior training on it, OTS maintained a high accuracy of 70.6%. This suggests that the "object-to-scene" logic transfers effectively across different camera types and environments.
  • Efficiency: The proposed Object Attention Block uses fewer parameters (1.1M) and fewer FLOPs (180.3M) than standard non-local blocks, making it suitable for real-time robotics applications.

Conclusion and Future Work

While OTS shows significant promise, it remains dependent on the accuracy of the underlying segmentation network. Failure to detect a crucial object can lead to misclassification. Future work will focus on integrating advanced data augmentation to diversify object features and expanding the number of object categories beyond the current 150 classes. Ultimately, the goal is to deploy OTS on mobile robots to enable real-time semantic mapping of complex human environments.

Summary updated

BibTeX
@inproceedings{miao2021object,
  author = {Bo Miao and Liguang Zhou and Ajmal Saeed Mian and Tin Lun Lam and Yangsheng Xu},
  title = {{Object-to-Scene: Learning to Transfer Object Knowledge to Indoor Scene Recognition}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636700},
}
Figure