Open Source Platform Digital Personal Assistant

Azat Khusnutdinov, Denis Usachev, Manuel Mazzara, Adil Khan, Ivan Panchenko

TL;DR

Researchers at Innopolis University have developed an open-source Digital Personal Assistant (DPA) designed as a transparent, extensible platform for engineers to test NLP hypotheses. By combining Word Movers Distance for intent recognition with an MQTT-based smart home architecture, the system provides a functional alternative to closed-source "black box" assistants like Alexa or Siri.

What Problem Does This Paper Address?

While Digital Personal Assistants (DPAs) have become ubiquitous, the market is dominated by proprietary giants like Google, Amazon, and Apple. For researchers and developers, these commercial products are "black boxes"—their internal logic is hidden, and their extensibility is limited by strict APIs.

The authors identify a significant gap: the lack of a flexible, open-source learning platform where engineers can modify the core NLP pipeline, experiment with different language models, and integrate custom hardware without jumping through corporate hoops. This paper introduces a prototype that bridges the gap between high-level Natural Language Processing (NLP) and low-level Internet of Things (IoT) control.

Methods

The authors built a modular ecosystem consisting of two primary components: the DPA core and a Smart Home automation module.

  • NLP Pipeline: The system uses the CoreNLP project for initial text processing (tokenization, Part-of-Speech tagging, and Named Entity Recognition).
  • Intent Recognition via WMD: Instead of simple keyword matching or complex deep learning models that require massive datasets, the authors utilized Word Movers Distance (WMD). WMD measures the "travel distance" between embedded words to find semantic similarity between a user's utterance and pre-defined intent samples. If WMD fails to find a match above a certain threshold, the system falls back to keyphrase matching.
  • Slot Filling: The DPA extracts parameters (dates, numbers, strings) to fill "slots" required by an intent. If information is missing, it triggers a "Dialog Support" module to ask the user follow-up questions.
  • IoT Architecture: The hardware side uses Arduino-compatible ESP8266 boards. Communication is handled via the MQTT protocol (using the Mosquitto broker), which is preferred over HTTP for its low power consumption and lightweight overhead in IoT environments.
  • Extensibility: 3rd-party developers can integrate new apps by providing a simple JSON manifest that defines intents, sample phrases, and required parameters.

Datasets

The paper does not utilize a large-scale public dataset for training. Instead, the system is designed to be "cold-started" by developers. Evaluation was performed using:

  • Internal Test Suite: A set of functional test cases (TC01-TC04) designed to verify query recognition, relay control, MQTT stability, and delegation logic.
  • Real-world Pilot: A week-long deployment in a residential room to test "up-time" and command reliability.

Key Insights and Ideas

  • Semantic vs. Lexical Matching: By using WMD, the system can understand that "I feel hot, limit heater till 20" and "Set up temperature at 20 degree" share the same intent, even if they share few common words. This is a significant step up from basic bag-of-words models.
  • Decoupling Logic from Execution: The DPA doesn't "know" how to turn off a light. It simply transforms natural language into a standardized JSON command. This allows the 3rd-party app (the Smart Home module) to remain fully autonomous and independently updatable.
  • The "Learning Platform" Philosophy: The authors emphasize that the goal isn't just to build a better Alexa, but to provide a "white box" where developers can see exactly why a command failed—whether it was a lemmatization error, a failed WMD match, or an MQTT timeout.

Key Results

The prototype demonstrated high reliability in a controlled environment:

  • Reliability: During a one-week real-world test, the system maintained 100% up-time and correctly executed all user commands.
  • Functional Validation: The system successfully passed all four core test cases, proving it could recognize queries, extract entities (like "the computer" or "16th of November"), and delegate those tasks to the hardware layer.
  • JSON Interoperability: The evaluation showed the DPA successfully generating complex JSON objects from natural language, correctly identifying the AppName, Intent, and Object (e.g., turning off "air conditioning" vs. "the light").

Limitations and Future Work

The authors are transparent about the prototype's current constraints:

  • Language Support: The initial language model is limited to English.
  • Slot Filling Constraints: The current version can only handle one value per data type per request (e.g., it might struggle with "Set the temperature to 20 and the humidity to 50" in a single sentence).
  • Dialog Complexity: While basic follow-up questions work, the system lacks a sophisticated state machine for complex, multi-turn conversations.
  • Future Directions: The team plans to introduce more language modules, improve data analysis for the smart home agent, and explore "proactive assistance" where the DPA predicts user needs based on sensor history.

Summary updated

Abstract

Nowadays Digital Personal Assistants (DPA) become more and more popular. DPAs help to increase quality of life especially for elderly or disabled people. In this paper we develop an open source DPA and smart home system as a 3- rd party extension to show the functionality of the assistant. The system is designed to use the DPA as a learning platform for engineers to provide them with the opportunity to create and test their own hypothesis. The DPA is able to recognize users' commands in natural language and transform it to the set of machine commands that can be used to control different 3rdparty application. We use smart home system as an example of such 3rd-party. We demonstrate that the system is able to control home appliances, like lights, or to display information about the current state of the home, like temperature, through a dialogue between a user and the Digital Personal Assistant.
BibTeX
@inproceedings{khusnutdinov2018open,
  author = {Azat Khusnutdinov and Denis Usachev and Manuel Mazzara and Adil Khan and Ivan Panchenko},
  title = {{Open Source Platform Digital Personal Assistant}},
  booktitle = {AINA Workshops},
  year = {2018},
  url = {https://openreview.net/forum?id=cfl-Vd_KZ8E},
}

Loading related content...

Figure