Multimodal Continuation-style Architectures for Human-Robot Interaction

Nikhil Krishnaswamy, James PustejovskyarXiv 2019 · arXiv

Abstract

This paper presents a novel architecture for human-robot interaction (HRI) that integrates real-time speech and gesture processing. By combining a modified pushdown automaton (PDA) with continuation-passing style (CPS) programming, the system enables virtual agents to incrementally process multimodal commands, resolve ambiguities through dialogue, and learn new gestures in a one-shot fashion.

Introduction

Interacting with a physical robot requires a departure from traditional text-based interfaces. Human communication is inherently multimodal, utilizing a complex combination of speech, deictic gestures (pointing), and body language. A robust HRI system must be able to hold incomplete instructions in memory, resolve spatial or linguistic ambiguities using multimodal cues, and execute composed commands in real time.

Traditional Finite State Automata (FSA) often fail in these scenarios due to the state explosion required to handle continuous real-world variables. We address this by proposing an architecture built on a modified PDA and CPS, allowing for flexible, incremental processing of situated dialogue.

Architecture and Methods

The proposed architecture is implemented in the VoxSim simulation environment. It treats speech and gesture as a unified ensemble where linguistic markers (e.g., "that") act as hooks for spatial data provided by gestures.

The VoxSim environment showing an agent asking a clarifying question to disambiguate a user's command.

Modified Pushdown Automaton (PDA)

To manage conversational context without the overhead of a massive FSA, we utilize a PDA. This model is modified to evaluate transitions based on conditions over continuous variables—such as checking if a 3D coordinate falls within a specific bounding box—rather than discrete symbol matches. Custom stack operations like FLUSH and POPUNTIL allow the system to clear or revert context during aborted or redirected actions.

Continuation-Passing Style (CPS)

CPS is employed to manage the "what to do next" logic of the interaction. When a user provides an incomplete command (e.g., "Put the cup..."), the action is pushed to the stack as a continuation—a function awaiting its arguments. This allows the system to enter a disambiguation loop to resolve the destination before executing the final action.

State transition diagrams illustrating the logic for disambiguating specific objects (left) and generalized arguments (right).

Key Insights and Results

The integration of CPS and a modified PDA yields several advantages:

  • Dialogue as Continuations: The system's internal logic mirrors human conversational patterns, effectively handling unresolved references and pauses.
  • One-Shot Gesture Learning: Because actions are functions waiting for arguments, the system can instantly map a novel iconic gesture to a specific object or action.
  • Complex Disambiguation: Qualitative results in VoxSim demonstrate the agent's ability to navigate multi-step loops, asking clarifying questions and processing multimodal feedback to resume original commands seamlessly.

Conclusion and Future Work

While the current system demonstrates high efficacy in a simulated environment, future work will focus on deployment on physical robotic platforms. This transition will require addressing the challenges of noisy sensor data and physical motor constraints not present in the current simulation.

Summary updated

Abstract

We present an architecture for integrating real-time, multimodal input into a computational agent's contextual model. Using a human-avatar interaction in a virtual world, we treat aligned gesture and speech as an ensemble where content may be communicated by either modality. With a modified nondeterministic pushdown automaton architecture, the computer system: (1) consumes input incrementally using continuation-passing style until it achieves sufficient understanding the user's aim; (2) constructs and asks questions where necessary using established contextual information; and (3) maintains track of prior discourse items using multimodal cues. This type of architecture supports special cases of pushdown and finite state automata as well as integrating outputs from machine learning models. We present examples of this architecture's use in multimodal one-shot learning interactions of novel gestures and live action composition.
BibTeX
@misc{krishnaswamy2019multimodal,
  author = {Nikhil Krishnaswamy and James Pustejovsky},
  title = {{Multimodal Continuation-style Architectures for Human-Robot Interaction}},
  year = {2019},
  eprint = {1909.08161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.08161},
}
Figure