Towards Real-world Deployment of NILM Systems: Challenges and Practices

Junyu Xue, Yu Zhang, Xudong Wang, Yi Wang, Guoming TangSustaincom 2024 · DOI · OpenReview

TL;DR

This paper tackles the real-world deployment challenges of Non-Intrusive Load Monitoring (NILM) systems by proposing a novel three-tier edge-cloud collaborative framework. By distributing computation between edge devices and the cloud, and employing a practical deployment strategy with Gunicorn and NGINX, they achieve high accuracy with significantly reduced latency and cloud workload.

What Problem Does This Paper Address?

Non-Intrusive Load Monitoring (NILM) is a promising technology for understanding household energy consumption without needing individual sensors for each appliance. This can lead to significant energy savings, better grid management, and fault detection. However, existing NILM research has largely focused on cloud-based solutions. These approaches often suffer from high computational costs, significant service delays due to data transmission and processing, and potential data privacy concerns. The paper aims to bridge the gap between theoretical NILM algorithms and practical, real-world deployment by addressing these limitations.

Methods

The core of the paper is a three-tier edge-cloud collaborative framework. This architecture distributes tasks across three levels:

  • Cloud Side: Hosts a powerful NILM server running a deep learning model (Seq2Point) for high-accuracy decomposition. This server is designed as a microservice for flexibility and scalability.
  • Edge Side: Employs a lightweight NILM model (XGBoost) for tasks like data preprocessing and potentially some local inference. This level utilizes terminal equipment (like smart meters or smart CT devices) and a system carrier for data handling.
  • Client Side: Represents the end-users who receive personalized insights and recommendations.

To facilitate efficient communication between the edge and cloud, the paper leverages RabbitMQ message queues. This acts as a buffer, handling high concurrent traffic and ensuring data is processed reliably.

For practical deployment, they integrate Gunicorn and NGINX. Gunicorn serves as a WSGI server to manage multiple worker processes, while NGINX acts as a reverse proxy, load balancer, and HTTP cache. This combination is crucial for handling high concurrency and ensuring responsiveness.

Datasets

The paper utilizes datasets collected from a laboratory environment designed to simulate a home-like setting. These datasets include:

  • Timestamp
  • Mains voltage, frequency, current transformer (CT) probe readings
  • Active power, mains current, reactive power, apparent power, power factor
  • Load labels for six types of electrical appliances: air purifier, heater, light bulb, fan, air compressor, and air conditioner.

The data was sampled every 2 seconds and was cleaned and relabeled for model training and evaluation. Two distinct datasets were used for training and evaluating edge-end and cloud-end models due to experimental environment changes.

Key Insights and Ideas

  • Edge-Cloud Synergy is Crucial: The paper strongly advocates that a hybrid approach, leveraging both edge and cloud capabilities, is essential for practical NILM deployment. The edge handles immediate processing and reduces latency, while the cloud provides the computational power for complex, accurate models.
  • Lightweight Edge Models are Viable: Using models like XGBoost on the edge can significantly reduce response times and cloud workload, even if they offer slightly lower accuracy than their cloud counterparts.
  • Practical Deployment Matters: Theoretical algorithms are insufficient. Integrating tools like Gunicorn and NGINX is vital for managing concurrency, load balancing, and ensuring a robust, responsive service in real-world scenarios.
  • RabbitMQ as a Communication Backbone: The use of RabbitMQ effectively decouples the edge and cloud, acting as a robust buffer for asynchronous data streams and high traffic volumes.
  • Trade-offs are Inherent: There's a clear trade-off between edge and cloud performance. Edge deployment offers speed, while cloud deployment offers accuracy. The proposed framework aims to balance these.

Key Results

  • Reduced Response Time: Edge-only deployment showed significantly lower response times compared to cloud-only deployment, especially under high concurrency. With 100 concurrent threads, edge response time was only 16.7% of the cloud's.
  • Improved Concurrency and Load Balancing: The integration of NGINX and Gunicorn dramatically improved concurrency handling. The combined system could handle up to 400 concurrent threads before experiencing failures, compared to around 200 for a basic Flask deployment.
  • High Decomposition Accuracy: The cloud-end Seq2Point model achieved high accuracy (average F1-score of 0.9411), while the edge-end XGBoost model provided reasonable accuracy (average F1-score of 0.7414) with much lower computational demands.
  • Data Preprocessing Benefits: Edge-side data preprocessing successfully filtered out approximately 1.8% of dirty data points, reducing transmission costs.

Limitations and Future Work

  • Laboratory Simulation: While aiming for real-world applicability, the experiments were conducted in a simulated home-like environment. Further validation in diverse, real-world settings is needed.
  • Appliance Specificity: The performance of the edge-end XGBoost model was noted to be lower for certain appliances (e.g., air purifiers, fans) due to data sparsity and the model's limited expressiveness.
  • Future Work: The authors suggest exploring personalized solutions for different households and iterative model updates, implying a need for continuous learning and adaptation within the framework. Further research into optimizing the division of labor between edge and cloud based on specific appliance characteristics and user needs would also be beneficial.

Summary updated

Abstract

Non-intrusive load monitoring (NILM), as a key load monitoring technology, can much reduce the deployment cost of traditional power sensors. Previous research has largely focused on developing cloud-exclusive NILM algorithms, which often result in high computation costs and significant service delays. To address these issues, we propose a three-tier framework to enhance the real-world applicability of NILM systems through edge-cloud collaboration. Considering the computational resources available at both the edge and cloud, we implement a lightweight NILM model at the edge and a deep learning based model at the cloud, respectively. In addition to the differential model implementations, we also design a NILM-specific deployment scheme that integrates Gunicorn and NGINX to bridge the gap between theoretical algorithms and practical applications. To verify the effectiveness of the proposed framework, we apply real-world NILM scenario settings and implement the entire process of data acquisition, model training, and system deployment. The results demonstrate that our framework can achieve high decomposition accuracy while significantly reducing the cloud workload and communication overhead under practical considerations.
BibTeX
@inproceedings{xue2024towards,
  author = {Junyu Xue and Yu Zhang and Xudong Wang and Yi Wang and Guoming Tang},
  title = {{Towards Real-world Deployment of NILM Systems: Challenges and Practices}},
  booktitle = {Sustaincom},
  year = {2024},
  doi = {10.1109/sustaincom63171.2024.00011},
}
Figure