Activity Prediction from Sensor Data using Convolutional Neural Networks and an Efficient Compression Method

Woojeong Jin, Dong-Jin Choi, Youngjin Kim, U KangJournal of KIISE 2018 · DOI

TL;DR

This paper tackles the challenge of accurately predicting occupant count and activities in smart buildings using low-cost sensors. They propose a novel deep learning approach combining infrared grid-eye sensor data with sound sensor data, achieving high accuracy. Furthermore, they introduce an efficient data compression method using Truncated SVD to reduce storage requirements without significant loss of performance.

What Problem Does This Paper Address?

Efficiently managing energy consumption in smart buildings hinges on understanding how many people are present and what they are doing. Traditional methods, like using cameras, are accurate but prohibitively expensive and generate massive amounts of data. This paper addresses the need for a more cost-effective and storage-efficient solution by leveraging readily available, low-cost sensors like infrared grid-eye sensors and sound sensors. The core problem is to accurately infer occupant count and activity types from these sensor streams while minimizing the storage footprint of the collected data.

Methods

The authors propose two main deep learning models based on Convolutional Neural Networks (CNNs), specifically leveraging the ResNet architecture.

  • 2D ResNet Model: This model combines data from an 8x8 infrared grid-eye sensor (which captures temperature variations and can be treated as image-like data) with sound sensor data. To balance the different data types and prevent the sound data from being ignored due to its lower dimensionality, the grid-eye sensor data is processed through a Deep Residual Network to extract refined features. These features are then concatenated with the sound sensor data and fed into fully connected layers for classification.
  • 3D ResNet Model: This model extends the 2D approach by incorporating temporal information. It takes sequential data from both the grid-eye and sound sensors as input, allowing the model to learn from the temporal dynamics of activities. This is achieved by using 3D convolutional kernels in the ResNet architecture.

For data compression, the paper utilizes Truncated Singular Value Decomposition (SVD). SVD is a matrix factorization technique. Truncated SVD approximates the original matrix by keeping only the most significant singular values and their corresponding vectors. This effectively reduces the dimensionality of the data, leading to significant storage savings while aiming to minimize the loss of information critical for accurate classification.

Datasets

The authors collected their own dataset for this study.

  • Data Sources: Infrared Grid-Eye sensor (8x8 array, measuring temperature) and a sound sensor (measuring ambient noise levels).
  • Data Collection Environment: The data was collected in a laboratory setting at Seoul National University, specifically in a room within the New Engineering Building.
  • Data Points: A total of 20,000 data points were collected, with each label (occupant count and activity type) having 5,000 data points.
  • Labels:
    • Occupant Count: 0, 1, 2, 3+ people.
  • Activity Types: 1. Absence, 2. Office work/Rest, 3. Conversation, 4. Walking, 5. Exercise.

Key Insights and Ideas

  • Sensor Fusion Benefits: Combining infrared grid-eye sensor data with sound sensor data significantly improves the accuracy of activity recognition compared to using the grid-eye sensor alone. This highlights the complementary nature of different sensor modalities.
  • Temporal Information is Crucial: Incorporating temporal data (recent sensor readings) through the 3D ResNet model leads to a substantial increase in accuracy for both occupant count and activity prediction. This suggests that understanding the sequence of events is vital for accurate activity inference.
  • Low-Cost Sensors are Viable: The study demonstrates that relatively inexpensive sensors can be effectively used for sophisticated tasks like activity prediction, making smart building solutions more accessible.
  • SVD for Efficient Compression: Truncated SVD proves to be an effective method for compressing sensor data, achieving significant storage reduction with minimal impact on classification accuracy. This is crucial for managing the vast amounts of data generated by IoT devices.
  • Sound Sensor Nuances: While sound sensors help, their effectiveness for distinguishing between different numbers of people can be limited if the sound patterns are not distinct (e.g., multiple people talking vs. one person talking).

Key Results

  • Occupant Count Prediction:
    • The 3D ResNet model achieved an accuracy of 95.3% for predicting the number of occupants.
  • The 2D ResNet model achieved an accuracy of 85.5% (Grid-Eye only) and 85.25% (Grid-Eye + Sound).
  • Activity Prediction:
    • The 3D ResNet model achieved an accuracy of 90.9% for predicting human activities.
  • The 2D ResNet model achieved an accuracy of 73.775% (Grid-Eye only) and 75.425% (Grid-Eye + Sound).
  • Data Compression:
    • Using Truncated SVD, the authors achieved a data compression rate of 6.25% (meaning the compressed data is 6.25% of the original size, or a reduction of 93.75%).
  • With this compression, the accuracy for occupant count dropped by 0-5%, and for activity type by 0-5% compared to the original data.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that the sound sensor's effectiveness in distinguishing between different numbers of people can be limited.
  • The accuracy improvements from temporal data plateaued after a certain number of consecutive data points (around 32-44), suggesting diminishing returns.
  • Future Work:
    • The paper implicitly suggests exploring more advanced sensor fusion techniques and potentially different deep learning architectures to further improve accuracy and robustness.
  • Investigating the impact of different compression ratios and exploring other compression algorithms could be future research directions.
  • The authors' work lays the groundwork for more sophisticated smart building management systems that can dynamically adapt to occupant presence and activities.

Summary updated

Abstract

건물 내 IoT 시스템을 이용한 재실자의 수 및 행동 유형 파악은 스마트 냉/난방 시스템 등에 사용되어 전력 효율을 개선시키고 비용 절감에 도움되는 중요한 문제이다. 실제 건물 관리 시스템에서는 카메라 등의 장비를 이용하여 방 안의 현황을 파악하고 재실자의 수와 행동 유형을 직접 파악하는 방법 등이 사용되고 있다. 이 방법으로 사람 수와 행동 유형을 파악하는 것은 비효율적일뿐만 아니라 데이터를 위한 방대한 저장 공간이 필요하다. 본 연구에서는 적외선 그리드 아이 센서와 소음 센서를 이용하여 실내 센서 데이터를 수집하였다. 또한 이 데이터를 토대로 재실자 수와 행동 유형을 파악하는 딥러닝 모델과 데이터의 시간적 특성을 고려하는 딥러닝 모델을 제안한다. 제안하는 모델은 약 95.3%의 정확도로 사람 수를 파악하고 90.9%의 정확도로 사람 행동 유형을 파악한다. 또한 Truncated SVD를 이용하여 정확도의 손실을 최소화하면서 저장 공간을 줄이는 방법을 제안한다.
BibTeX
@article{jin2018activity,
  author = {Woojeong Jin and Dong-Jin Choi and Youngjin Kim and U Kang},
  title = {{Activity Prediction from Sensor Data using Convolutional Neural Networks and an Efficient Compression Method}},
  journal = {Journal of KIISE},
  year = {2018},
  doi = {10.5626/jok.2018.45.6.564},
}
Figure