Open source platform Digital Personal Assistant

Denis Usachev, Azat Khusnutdinov, Manuel Mazzara, Adil Khan, Ivan Panchenko

Abstract

Nowadays Digital Personal Assistants (DPA) become more and more popular. DPAs help to increase quality of life especially for elderly or disabled people. In this paper we develop an open source DPA and smart home system as a 3rd-party extension to show the functionality of the assistant. The system is designed to use the DPA as a learning platform for engineers to provide them with the opportunity to create and test their own hypothesis. The DPA is able to recognize users' commands in natural language and transform it to the set of machine commands that can be used to control different 3rd-party application. We use smart home system as an example of such 3rd-party. We demonstrate that the system is able to control home appliances, like lights, or to display information about the current state of the home, like temperature, through a dialogue between a user and the Digital Personal Assistant.

Introduction

While Digital Personal Assistants (DPAs) have become ubiquitous in modern households, the market is currently dominated by proprietary "black box" systems such as Amazon Alexa, Apple Siri, and Google Assistant. For researchers and developers, these commercial products present significant hurdles: their internal logic is opaque, and extensibility is often restricted by rigid APIs. This lack of transparency limits the ability of engineers to experiment with novel Natural Language Processing (NLP) models or custom hardware integrations.

To address this gap, we present an open-source DPA designed as a flexible learning platform. This system bridges the gap between high-level NLP and low-level Internet of Things (IoT) control, allowing developers to modify the core pipeline and test hypotheses in a real-world environment.

System Architecture and Methodology

The system is built upon a modular ecosystem consisting of a DPA core and a Smart Home automation module. The high-level architecture facilitates communication between the internet, a central server, and edge devices.

High-level architecture of the DPA and Smart Home ecosystem.

NLP Pipeline and Intent Recognition

The DPA core utilizes the CoreNLP project for initial text processing tasks, including tokenization and Named Entity Recognition. For intent recognition, we implemented Word Movers Distance (WMD). Unlike simple keyword matching, WMD measures the semantic similarity between a user's utterance and pre-defined intent samples by calculating the "travel distance" between embedded words. This allows the system to understand varied phrasing, such as recognizing that "I feel hot" and "Set the temperature to 20" may share the same underlying intent.

Dialog Support and Slot Filling

To execute commands, the DPA must extract specific parameters (slots). If a user's initial query is incomplete, a "Dialog Support" module triggers follow-up questions to gather the missing information, ensuring robust interaction.

IoT and Communication Protocol

The hardware implementation utilizes Arduino-compatible ESP8266 boards to interface with sensors and actuators. To ensure low power consumption and lightweight overhead, the system employs the MQTT protocol via a Mosquitto broker.

MQTT sequence diagram showing communication between the ESP8266, Broker, and Home Assistant.

As shown in the sequence diagram, the ESP8266 and the Home Assistant application both connect to the MQTT broker. The ESP8266 subscribes to command topics and publishes sensor data, while the Home Assistant manages the logic for processing these messages and issuing new commands. This decoupling ensures that the 3rd-party application remains autonomous and easily updatable.

Results and Discussion

The prototype was evaluated through a series of functional test cases and a week-long residential pilot. The system demonstrated high reliability, maintaining 100% uptime and correctly executing all user commands during the test period.

Key findings include:

  • Semantic Accuracy: The WMD-based approach successfully mapped diverse natural language inputs to standardized JSON commands.
  • Interoperability: The system proved capable of generating complex JSON objects that correctly identified application names, intents, and target objects (e.g., distinguishing between "the light" and "air conditioning").
  • Transparency: As a "white box" platform, the system allowed developers to pinpoint specific failure points, whether they occurred during lemmatization, semantic matching, or hardware communication.

Conclusion

This open-source DPA provides a functional alternative to proprietary assistants, serving as a robust platform for engineering education and NLP research. Future work will focus on expanding language support beyond English, improving multi-value slot filling, and developing proactive assistance features that leverage sensor history to predict user needs.

Summary updated

Abstract

Nowadays Digital Personal Assistants (DPA) become more and more popular. DPAs help to increase quality of life especially for elderly or disabled people. In this paper we develop an open source DPA and smart home system as a 3-rd party extension to show the functionality of the assistant. The system is designed to use the DPA as a learning platform for engineers to provide them with the opportunity to create and test their own hypothesis. The DPA is able to recognize users' commands in natural language and transform it to the set of machine commands that can be used to control different 3rd-party application. We use smart home system as an example of such 3rd-party. We demonstrate that the system is able to control home appliances, like lights, or to display information about the current state of the home, like temperature, through a dialogue between a user and the Digital Personal Assistant.
BibTeX
@inproceedings{usachev2018open,
  author = {Denis Usachev and Azat Khusnutdinov and Manuel Mazzara and Adil Khan and Ivan Panchenko},
  title = {{Open source platform Digital Personal Assistant}},
  booktitle = {AINA Workshops},
  year = {2018},
  url = {https://openreview.net/forum?id=cfl-Vd_KZ8E},
}
Figure