A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces

Jan Zacharias, Michael Barz, Daniel SonntagarXiv 2018 · arXiv

Introduction

Intelligent User Interfaces (IUIs) represent the frontier of human-computer interaction, aiming to create systems that understand multimodal inputs such as speech, gaze, and gestures. While deep learning (DL) has provided the "intelligence" required for these tasks, developers face significant hurdles. The "intelligence" is often narrow, and the sheer volume of available open-source libraries makes it difficult to select the right tools for specific components like dialogue management or intent extraction. This paper provides a structured roadmap by mapping the DL software landscape to a functional IUI architecture.

A Functional Architecture for IUIs

To organize the diverse array of tools, the authors utilize a conceptual architecture based on the work of Wahlster and Maybury. This framework categorizes IUI components into four primary functional blocks: Input Processing & Media Analysis, Interaction Management, Output Rendering & Media Design, and Backend Services.

Functional architecture for Intelligent User Interfaces, categorizing components into processing, management, and rendering blocks.

By grouping tools into these blocks, researchers can better identify gaps in current software and select libraries that are interoperable within a standard pipeline. For instance, Natural Language Understanding (NLU) falls under Media Analysis, while dialogue flow is handled by Interaction Management. The architecture also emphasizes the importance of a shared representation layer for user models, discourse history, and domain knowledge.

The Landscape of Deep Learning Toolkits

A core contribution of this survey is the quantitative evaluation of general-purpose DL frameworks. The authors proposed a "Popularity Rating" based on weighted GitHub metrics including stars, forks, and active contributions to help developers gauge community support and longevity.

Comparison of popular deep learning frameworks based on GitHub metrics, licenses, and supported languages.

As of the study's conclusion, TensorFlow emerged as the dominant leader with a 100% rating, followed by Keras and Caffe. Beyond general frameworks, the paper highlights specialized tools like RASA_NLU for dialogue management and NVIDIA’s DIGITS for interactive vision model training. The authors note that while general frameworks are robust, specialized academic tools often suffer from "bit rot"—technical debt caused by outdated dependencies and lack of documentation.

Key Insights: From Static Models to Interactive ML

The survey identifies several critical trends and pitfalls in the current ecosystem:

  • Interactive Machine Learning (IML): The authors advocate for a shift toward "human-in-the-loop" systems. Rather than training static models, IUIs should allow users to "teach" the system in real-time, enhancing personalization and robustness in "open world" settings.
  • The Role of Containerization: Tools that utilize Docker, such as RASA, are significantly easier to adopt because they mitigate the "installation hell" associated with complex dependency chains.
  • Deep but Narrow: Current AI often achieves superhuman performance in controlled environments but fails dangerously in others, necessitating "lifelong learning" models.

Conclusion and Future Directions

The future of IUIs lies in moving from simple algorithmic improvements to "Machine Teaching." This paradigm focuses on the teacher's interface, allowing domain experts—rather than just AI specialists—to build and refine intelligent systems without needing deep technical knowledge of underlying parameters. Addressing the reproducibility crisis in academic software remains a primary challenge for the next generation of intelligent interfaces.

Summary updated

Abstract

This paper provides an overview of prominent deep learning toolkits and, in particular, reports on recent publications that contributed open source software for implementing tasks that are common in intelligent user interfaces (IUI). We provide a scientific reference for researchers and software engineers who plan to utilise deep learning techniques within their IUI research and development projects.
BibTeX
@misc{zacharias2018survey,
  author = {Jan Zacharias and Michael Barz and Daniel Sonntag},
  title = {{A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces}},
  year = {2018},
  eprint = {1803.04818},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.04818},
}
Figure