Helion: Enabling Natural Testing of Smart Homes

Prianka Mandal, Sunil Manandhar, Kaushal Kafle, Kevin Moran, Denys Poshyvanyk, Adwait Nadkarni

TL;DR

Helion addresses a major bottleneck in smart home security: the lack of realistic testing data. By applying n-gram language models to user-defined automation routines, Helion generates "natural" event sequences that mimic real-world home usage, allowing researchers to move beyond testing with random, nonsensical event permutations.

What Problem Does This Paper Address?

As smart homes become more complex, ensuring their security and safety is critical. However, the systems designed to find bugs or security flaws in home automation need realistic "test cases" to be effective.

Currently, researchers often rely on two flawed methods:

  1. Random Permutations: Generating random sequences of events (e.g., "Turn on oven," "Unlock front door," "Open blinds"). These rarely reflect how humans actually live and can lead to impractical security designs.
  2. Real Execution Traces: Collecting actual data from users' homes. This is extremely privacy-invasive and often contains "noise"—device-specific technical glitches that obscure the actual semantic usage of the home.

The authors identify a need for synthetically generated but realistic scenarios—sequences of events that could happen in a real home but are generated in a controlled, privacy-preserving way.

Methods

Helion treats smart home automation like a language. It builds on the "naturalness of code" hypothesis, suggesting that just as human-written software has predictable patterns, so do the routines humans create for their homes.

The framework follows a four-step pipeline:

  1. Data Collection: Instead of raw traces, they collect "routines" (IF-THEN rules) and "execution indicators" (metadata about when and how often these rules run).
  2. Informed Scheduling: This is a novel step where Helion uses the execution indicators to transform static rules into a chronological sequence of events (e.g., mapping a "morning" routine to a specific time slot in a simulated month).
  3. N-gram Language Modeling: They use statistical n-gram models ($n \ge 3$) to learn the regularities of these sequences. By assuming the Markov property, the model predicts the next event based on a short history of previous events.
  4. Scenario Prediction: The model can generate two types of scenarios:
  5. "Up" Flavor (Natural): Predicts the most probable next events to create realistic test cases.
  6. "Down" Flavor (Unrealistic): Predicts the least probable events. This is specifically useful for "negative testing" to see how a system handles rare, potentially dangerous edge cases.

The authors also introduced HelionHA, an extension for the popular open-source Home Assistant platform, which allows these predicted scenarios to be executed on real or virtual smart devices.

Datasets

The authors created the HOME Corpus:

  • Source: A user survey of 40 participants.
  • Content: 273 routines (233 unique) expressed in plain English and then tokenized.
  • Scale: Using informed scheduling, they expanded these routines into 40 month-long event sequences, totaling 30,518 events.
  • Availability: The anonymized dataset and code are publicly available on GitHub.

Key Insights and Ideas

  • Routines as the "Source Code" of a Home: Rather than looking at raw sensor data, Helion looks at the intent of the user (the routines they program). This filters out technical noise and focuses on semantic behavior.
  • Contextual Depth ($n \ge 3$): The authors found that using $n < 3$ (unigrams or bigrams) ignores the context necessary to capture meaningful home workflows. A sequence like "Motion Detected $\rightarrow$ Lights On $\rightarrow$ Door Unlocked" requires at least a 3-gram to maintain its semantic logic.
  • The "Down" Flavor for Security: In security testing, the most likely path isn't always the most interesting. By specifically generating unlikely (but still syntactically valid) sequences, Helion helps researchers find safety violations that only occur during rare "perfect storm" conditions.

Key Results

  • System Integration: The paper demonstrates a functional end-to-end pipeline where a researcher can pick a model (e.g., a 4-gram model), choose a "flavor," and watch Home Assistant execute the resulting test case on a dashboard.
  • Token-to-Entity Mapping: The authors successfully implemented a script (parse_token.py) that translates abstract model tokens (like <light, switch, ON>) into actionable Home Assistant commands for specific devices.
  • Practical Utility: HelionHA provides a UI that allows users to monitor smart home states in real-time as a scenario executes, bridging the gap between abstract statistical modeling and physical device testing.

Limitations and Future Work

  • Dataset Size: While 40 users provided a solid foundation, the "vocabulary" of smart home events is vast. Larger datasets would be needed to capture more diverse living patterns.
  • Model Complexity: The current version uses n-grams, which are computationally efficient but have a limited "memory." Future work might explore more advanced architectures like Transformers (e.g., GPT-style models) to capture longer-range dependencies in home behavior.
  • User Behavior Shifts: The model assumes routines are relatively static. It does not yet account for how a user’s behavior might change seasonally or over long periods.

Summary updated

Abstract

Prior work has developed numerous systems that test the security and safety of smart homes. For these systems to be applicable in practice, it is necessary to test them with realistic scenarios that represent the use of the smart home, i.e., home automation, in the wild. This demo paper presents the technical details and usage of Helion, a system that uses n-gram language modeling to learn the regularities in user-driven programs, i.e., routines developed for the smart home, and predicts natural scenarios of home automation, i.e., event sequences that reflect realistic home automation usage. We demonstrate the HelionHA platform, developed by integrating Helion with the popular Home Assistant smart home platform. HelionHA allows an end-to-end exploration of Helion's scenarios by executing them as test cases with real and virtual smart home devices.
BibTeX
@inproceedings{mandal2023helion,
  author = {Prianka Mandal and Sunil Manandhar and Kaushal Kafle and Kevin Moran and Denys Poshyvanyk and Adwait Nadkarni},
  title = {{Helion: Enabling Natural Testing of Smart Homes}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2023},
  url = {https://openreview.net/forum?id=f6LlgxhBxu},
}

Loading related content...

Figure