Zero-Shot Visual Slot Filling as Question Answering

Larry Heck, Simon HeckarXiv 2020 · arXiv

Abstract

Digital assistants struggle to scale to the vast ecosystem of third-party mobile applications due to the high cost of manual slot labeling. This paper presents mForms, a multimodal framework that reformulates mobile form-filling as an extractive Question Answering (QA) task. By leveraging visual semantic extraction and natural language question generation, mForms enables zero-shot slot filling across diverse app domains without requiring app-specific training data.

Introduction

Traditional digital assistants rely on "slot filling" to interact with apps. This process typically requires developers to provide thousands of labeled examples to map user utterances to specific form fields (e.g., mapping "Toyota Prius" to a vehicle_type tag). Such an approach is inherently unscalable for the "long tail" of millions of mobile apps. The authors propose moving away from treating form fields as abstract symbols and instead treating them as semantic entities that can be understood through visual context.

The mForms Framework

The mForms system bridges the gap between visual user interfaces and spoken dialogue through a three-stage pipeline:

  1. Visual Semantic Extraction: A Convolutional Neural Network (CNN) identifies UI components (e.g., checkboxes), text buttons, and icon classes (e.g., a "bed" icon for hotel nights) directly from the screen.
  2. Question Generation: Identified elements are converted into natural language questions using rule-based templates. For instance, a field labeled "Odometer Value" becomes "What is the odometer value?"
  3. Extractive QA: The user's request is treated as a context paragraph. Both the generated question and the user's utterance are fed into a transformer model to identify the specific text span that answers the question.
The mForms pipeline: translating UI elements into questions to extract information from user utterances to fill a mobile form.

Model Architecture

At the core of mForms is a transformer-based model (such as ALBERT) pre-trained on large-scale QA datasets like SQuAD. The model takes a concatenated input consisting of the generated "Slot Question" and the user's "Utterance." It then predicts the start and end spans within the utterance that answer the question. This architecture allows the system to handle "unanswerable" questions gracefully, rejecting slots not mentioned by the user rather than forcing an incorrect match.

The mForms model architecture, utilizing a transformer-based extractive QA approach to map questions and utterances to specific text spans.

Key Results and Insights

Evaluation on the new mForms dataset and the classic ATIS dataset demonstrates significant improvements in data efficiency and zero-shot performance:

  • Zero-Shot Success: In scenarios with no prior training data for a specific app, mForms achieved an F1 score of 0.48, whereas traditional baseline models scored 0.00.
  • Extreme Data Efficiency: On the ATIS dataset, mForms reached a state-of-the-art F1 score of 0.97 using only 500 training samples, compared to the 4,400+ samples typically required by traditional models.
  • Cross-Domain Learning: Fine-tuning on one domain (e.g., flights) improved performance on unrelated domains (e.g., vehicle logging), proving the model learns general concept extraction.

Conclusion and Future Work

By treating UI elements as semantic questions rather than arbitrary tags, mForms provides a scalable path for digital assistants to interact with any mobile application. While currently reliant on rule-based templates, future work will explore the use of Large Language Models (LLMs) for more flexible question generation and handling high-density, complex interfaces.

Summary updated

Abstract

This paper presents a new approach to form-filling by reformulating the task as multimodal natural language Question Answering (QA). The reformulation is achieved by first translating the elements on the GUI form (text fields, buttons, icons, etc.) to natural language questions, where these questions capture the element's multimodal semantics. After a match is determined between the form element (Question) and the user utterance (Answer), the form element is filled through a pre-trained extractive QA system. By leveraging pre-trained QA models and not requiring form-specific training, this approach to form-filling is zero-shot. The paper also presents an approach to further refine the form-filling by using multi-task training to incorporate a potentially large number of successive tasks. Finally, the paper introduces a multimodal natural language form-filling dataset Multimodal Forms (mForms), as well as a multimodal extension of the popular ATIS dataset to support future research and experimentation. Results show the new approach not only maintains robust accuracy for sparse training conditions but achieves state-of-the-art F1 of 0.97 on ATIS with approximately 1/10th of the training data.
BibTeX
@misc{heck2020zero,
  author = {Larry Heck and Simon Heck},
  title = {{Zero-Shot Visual Slot Filling as Question Answering}},
  year = {2020},
  eprint = {2011.12340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.12340},
}
Figure