Poster Abstract: Intuitive Appliance Identification using Image Matching in Smart Buildings

Kaifei Chen, John Kolb, Jonathan Fürst, Dezhi Hong, Randy H. KatzBuildSys 2015 · DOI

TL;DR

This paper introduces a vision-based system for intuitively identifying smart appliances in commercial buildings by matching real-time camera images to a pre-built 3D model of the environment. It achieves high accuracy in appliance identification, offering a more user-friendly alternative to traditional, cumbersome methods.

What Problem Does This Paper Address?

As smart buildings become increasingly populated with interconnected appliances, users face a growing challenge in identifying and interacting with specific devices through software. Existing methods often require users to describe appliances using rigid syntax or necessitate the deployment of additional, complex infrastructure like infrared transceivers. This paper addresses the need for a more intuitive and less intrusive way for users to locate and control smart appliances in their environment.

Methods

The proposed system operates in two phases:

  1. Modeling Phase: This phase involves creating a visual 3D model of the building's interior. Structure from Motion (SfM) is used to reconstruct a point cloud from RGB-D video captured by a building manager. This point cloud is then annotated with labels for each appliance.
  2. Matching Phase: In this phase, a user's smartphone captures an RGB image of the environment. The system then registers this image onto the existing 3D model. This is achieved by first finding the most similar registered image from the model and then using a Perspective-N-Point (PnP) algorithm to compute the camera's location and orientation. Finally, visible appliances are identified by projecting their labeled 3D points onto the 2D image plane.

Datasets

  • RTAB-Map Multi-session Dataset: This dataset consists of five videos of a building floor collected by a robot with a Microsoft Kinect. For evaluation, one video was used for modeling (labeling 10 objects), and 46 RGB images from other videos containing labeled objects were used for testing.
  • Lab Deployment Dataset: This dataset was collected by a human using a Microsoft Kinect in a lab environment, comprising 1075 images and 15 labeled objects. The evaluation involved taking multiple pictures of each object from different angles after a few days.

Key Insights and Ideas

  • "What you see is what you interact with": The core idea is to leverage computer vision to create a seamless and intuitive user experience for appliance interaction, mirroring how humans naturally perceive and engage with their surroundings.
  • Vision-based approach reduces deployment overhead: By relying on image matching and a pre-built 3D model, the system avoids the need for extensive, device-specific hardware installations like infrared transceivers.
  • SfM for 3D reconstruction and PnP for real-time matching: The paper effectively combines established computer vision techniques (SfM for scene understanding and PnP for pose estimation) in a novel pipeline for appliance identification.
  • Intuitive user interaction is paramount: The system prioritizes ease of use, recognizing that complex query methods or additional hardware hinder adoption in smart building environments.

Key Results

  • The system achieved 98% accuracy on the RTAB-Map multi-session dataset and 100% recall and precision on a public robot-collected dataset.
  • On the RTAB-Map dataset, the system achieved 100% success when considering categories 4-8 (successful identification, including partially visible or occluded objects) as successful.
  • For the lab deployment dataset, 730 out of 810 instances were categorized as successful (types 4-8), which the authors consider promising for real-world deployment despite some noise from environmental changes.

Limitations and Future Work

  • Occlusion analysis: The paper acknowledges that the system does not explicitly perform occlusion analysis, which leads to some instances being categorized as "Occluded, Identified" (type 6) when they are visible but not fully captured by the model's labeling.
  • Partial visibility handling: While the system can handle partially visible objects (types 4 and 5), it might not identify them if the labeled points are not within the viewing cone. The authors note this is acceptable as users intuitively capture the entire target object.
  • Future work: The paper suggests that real-world deployments are feasible and that the current results are promising. Further work could focus on improving robustness to environmental changes and more sophisticated occlusion handling.

Summary updated

BibTeX
@inproceedings{chen2015poster,
  author = {Kaifei Chen and John Kolb and Jonathan Fürst and Dezhi Hong and Randy H. Katz},
  title = {{Poster Abstract: Intuitive Appliance Identification using Image Matching in Smart Buildings}},
  booktitle = {BuildSys},
  year = {2015},
  doi = {10.1145/2821650.2830299},
}
Figure