Helion: Enabling Natural Testing of Smart Homes

Prianka Mandal, Sunil Manandhar, Kaushal Kafle, Kevin Moran, Denys Poshyvanyk, Adwait Nadkarni

Abstract

Prior research has produced numerous systems aimed at securing smart home environments. However, the practical efficacy of these systems depends on their ability to be tested against realistic scenarios that reflect actual human behavior. This paper presents Helion, a system that leverages n-gram language modeling to learn the regularities of user-defined smart home routines. By treating home automation as a structured language, Helion predicts "natural" event sequences that mimic real-world usage. We further introduce HelionHA, an integration with the Home Assistant platform that enables the end-to-end execution of these generated scenarios as test cases for real and virtual smart home devices.

Introduction

As smart home ecosystems grow in complexity, ensuring their safety and security has become a critical challenge. Security researchers typically evaluate smart home defense systems using one of two methods: random event permutations or real execution traces. Both approaches are fundamentally flawed. Random sequences (e.g., unlocking a door immediately followed by opening the blinds and turning on an oven) rarely reflect human logic, leading to impractical security evaluations. Conversely, collecting real execution traces is highly privacy-invasive and often contains "noise"—device-specific technical glitches that obscure the semantic intent of the user.

Helion addresses this bottleneck by generating synthetically realistic scenarios. By focusing on the "routines" (IF-THEN rules) users program into their homes, Helion captures the underlying intent of home automation while preserving privacy and filtering out technical noise.

The Helion Framework

Helion is built on the "naturalness of code" hypothesis, which suggests that just as human-written software follows predictable patterns, so do the automation routines created for smart homes. The framework utilizes a four-step pipeline to transform abstract user intent into executable test cases.

1. Data Collection and the HOME Corpus

To train its models, the authors developed the HOME Corpus. They surveyed 40 participants to collect 273 routines (233 unique) expressed in plain English. These routines were then tokenized into a standardized format representing device types, attributes, and states.

2. Informed Scheduling

Static routines do not inherently contain the chronological data needed for testing. Helion employs a novel "Informed Scheduling" step, using execution indicators (metadata regarding how often and when rules typically run) to expand these routines into a month-long sequence of events. For the HOME Corpus, this resulted in a dataset of over 30,000 events.

3. N-gram Language Modeling

Helion uses statistical n-gram models ($n \ge 3$) to learn the regularities within these event sequences. By assuming the Markov property—where the probability of an event depends on a fixed window of preceding events—the model learns to predict the next logical step in a home automation sequence.

4. Scenario Prediction: "Up" and "Down" Flavors

Helion can generate two distinct types of scenarios based on the model's probability distributions:

  • "Up" Flavor (Natural): The model predicts the most probable next events. These are used for "positive testing" to ensure security systems do not interfere with normal, everyday home usage.
  • "Down" Flavor (Unrealistic): The model predicts the least probable (but still syntactically valid) events. This is a powerful tool for "negative testing," allowing researchers to discover safety violations that occur during rare, unexpected "perfect storm" conditions.

HelionHA: From Theory to Execution

To bridge the gap between statistical modeling and physical testing, the authors developed HelionHA. This extension for the open-source Home Assistant platform allows researchers to execute Helion’s predicted scenarios on actual hardware or virtualized device instances.

A key component of this integration is the parse_token.py script, which maps abstract model tokens (e.g., <light, switch, ON>) to specific entity IDs within a Home Assistant configuration. This allows for a real-time, end-to-end exploration of scenarios where researchers can monitor device states on a dashboard as the Helion-generated test case unfolds.

Key Insights

  • The Importance of Context: The authors found that models with $n < 3$ (unigrams or bigrams) fail to capture the semantic logic of a home. For example, a sequence like "Motion Detected $\rightarrow$ Lights On $\rightarrow$ Door Unlocked" requires at least a 3-gram to maintain the necessary context for a realistic workflow.
  • Routines as Semantic Source Code: By training on user-defined routines rather than raw sensor data, Helion focuses on the intent of the smart home. This approach effectively bypasses the noise of hardware-level events and focuses on the high-level logic where security vulnerabilities often reside.

Conclusion and Future Work

Helion provides a scalable, privacy-preserving method for generating realistic smart home test data. While the current implementation uses n-gram models for their computational efficiency, future iterations may explore Transformer-based architectures to capture even longer-range dependencies in user behavior. Additionally, expanding the HOME Corpus will be essential to capturing the vast and evolving vocabulary of modern smart home devices. By providing both the dataset and the HelionHA platform to the community, the authors have established a new foundation for the rigorous testing of smart home security systems.

Summary updated

Abstract

Prior work has developed numerous systems that test the security and safety of smart homes. For these systems to be applicable in practice, it is necessary to test them with realistic scenarios that represent the use of the smart home, i.e., home automation, in the wild. This demo paper presents the technical details and usage of Helion, a system that uses n-gram language modeling to learn the regularities in user-driven programs, i.e., routines developed for the smart home, and predicts natural scenarios of home automation, i.e., event sequences that reflect realistic home automation usage. We demonstrate the HelionHA platform, developed by integrating Helion with the popular Home Assistant smart home platform. HelionHA allows an end-to-end exploration of Helion's scenarios by executing them as test cases with real and virtual smart home devices.
BibTeX
@inproceedings{mandal2023helion,
  author = {Prianka Mandal and Sunil Manandhar and Kaushal Kafle and Kevin Moran and Denys Poshyvanyk and Adwait Nadkarni},
  title = {{Helion: Enabling Natural Testing of Smart Homes}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2023},
  url = {https://openreview.net/forum?id=f6LlgxhBxu},
}
Figure