Multi-modal User Intent Classification Under the Scenario of Smart Factory (Student Abstract)

Yu-Ching Chiu, Bo-Hao Chang, Tzu-Yu Chen, Cheng-Fu Yang, Nanyi Bi, Richard Tzong-Han Tsai, Hung-yi Lee, Jane Yung-jen HsuAAAI 2021 · OpenReview

Abstract

Question-answering systems are becoming increasingly popular in Natural Language Processing, especially when applied in smart factory settings. A common practice in designing those systems is through intent classification. However, in a multiple-stage task commonly seen in those settings, relying solely on intent classification may lead to erroneous answers, as questions rising from different work stages may share the same intent but have different contexts and therefore require different answers. To address this problem, we designed an interactive dialogue system that utilizes contextual information to assist intent classification in a multiple-stage task. Specifically, our system incorporates user’s utterances with real-time video feed to better situate users’ questions and analyze their intent.

Introduction: The Problem of Contextual Ambiguity

In the complex environment of a smart factory, standard question-answering (QA) systems often fail because they rely exclusively on text or speech. For example, a worker asking "Where does this go?" requires a different answer depending on whether they are holding a robot's arm or its leg. This highlights two primary hurdles in industrial human-AI interaction: verbalization difficulty, where users lack technical vocabulary for specific components, and contextual ambiguity, where identical intents require different solutions based on the current assembly stage.

Example of intent classification mapping with visual context.

Without visual context, a text-only model might confuse these intents, leading to irrelevant or even dangerous instructions. To solve this, we propose a system that "sees" the physical context of the assembly task to ground abstract language in reality.

Multimodal System Architecture

Our system utilizes a multi-step pipeline to fuse linguistic and visual data for accurate intent classification. The architecture consists of two main branches that converge into a final classifier.

Multimodal system architecture for intent classification.

Text Processing

User speech is converted to text via Automatic Speech Recognition (ASR). This text is then processed by BERT (Bidirectional Encoder Representations from Transformers) to generate a high-dimensional representation vector of the query, capturing the semantic meaning of the user's question.

Visual State Discriminative Model (V-SDM)

The visual component uses the YOLOv3 (You Only Look Once) object detection network to identify robot components (e.g., head, neck, body) in the video feed. To ensure robustness against motion blur or temporary occlusions, the system employs Temporal Accumulation. Instead of a single frame, the V-SDM accumulates probability vectors across a video clip. If a component's presence exceeds a threshold, it is marked as "present" (1) in a binary object vector.

Multimodal Fusion

The BERT text vector and the V-SDM object vector are concatenated and fed into a Multi-Layer Perceptron (MLP) classifier. This classifier predicts the final intent class, allowing the system to provide a context-aware response that accounts for the specific part the user is handling.

Dataset Construction

As no standard dataset existed for this specific scenario, we curated a custom dataset focused on the assembly of a Meccanoid Robot:

  1. Wizard-of-Oz Pilot Study: A human "wizard" simulated the AI to identify 21 core intent scenarios from real user interactions.
  2. User Question Collection: Using Amazon Mechanical Turk, we generated 3,769 question variations based on the core intents.
  3. Visual Data: We captured and augmented 447 images of robot components to train the YOLOv3 model.
  4. Evaluation Set: A test set of 798 question variations paired with corresponding video clips was used to validate the system.

Conclusion and Future Work

This research demonstrates that the "state of the workbench" is a vital feature for Natural Language Understanding in industrial settings. By treating the presence of physical objects as a feature vector, the system effectively bridges the verbalization gap, allowing users to use deictic expressions like "this" or "that."

Future work will focus on expanding the visual dataset to improve generalization across lighting conditions and moving beyond binary state representations to include granular spatial information, such as the orientation and precise location of components.

Summary updated

Abstract

Question-answering systems are becoming increasingly popular in Natural Language Processing, especially when applied in smart factory settings. A common practice in designing those systems is through intent classification. However, in a multiple-stage task commonly seen in those settings, relying solely on intent classification may lead to erroneous answers, as questions rising from different work stages may share the same intent but have different contexts and therefore require different answers. To address this problem, we designed an interactive dialogue system that utilizes contextual information to assist intent classification in a multiple-stage task. Specifically, our system incorporates user’s utterances with real-time video feed to better situate users’ questions and analyze their intent.
BibTeX
@inproceedings{chiu2021multi,
  author = {Yu-Ching Chiu and Bo-Hao Chang and Tzu-Yu Chen and Cheng-Fu Yang and Nanyi Bi and Richard Tzong-Han Tsai and Hung-yi Lee and Jane Yung-jen Hsu},
  title = {{Multi-modal User Intent Classification Under the Scenario of Smart Factory (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=spsZjm8aOM2},
}
Figure