Abstract
This paper presents a novel architecture for human-robot interaction (HRI) that integrates real-time speech and gesture processing. By combining a modified pushdown automaton (PDA) with continuation-passing style (CPS) programming, the system enables virtual agents to incrementally process multimodal commands, resolve ambiguities through dialogue, and learn new gestures in a one-shot fashion.
Introduction
Interacting with a physical robot requires a departure from traditional text-based interfaces. Human communication is inherently multimodal, utilizing a complex combination of speech, deictic gestures (pointing), and body language. A robust HRI system must be able to hold incomplete instructions in memory, resolve spatial or linguistic ambiguities using multimodal cues, and execute composed commands in real time.
Traditional Finite State Automata (FSA) often fail in these scenarios due to the state explosion required to handle continuous real-world variables. We address this by proposing an architecture built on a modified PDA and CPS, allowing for flexible, incremental processing of situated dialogue.
Architecture and Methods
The proposed architecture is implemented in the VoxSim simulation environment. It treats speech and gesture as a unified ensemble where linguistic markers (e.g., "that") act as hooks for spatial data provided by gestures.
Modified Pushdown Automaton (PDA)
To manage conversational context without the overhead of a massive FSA, we utilize a PDA. This model is modified to evaluate transitions based on conditions over continuous variables—such as checking if a 3D coordinate falls within a specific bounding box—rather than discrete symbol matches. Custom stack operations like FLUSH and POPUNTIL allow the system to clear or revert context during aborted or redirected actions.
Continuation-Passing Style (CPS)
CPS is employed to manage the "what to do next" logic of the interaction. When a user provides an incomplete command (e.g., "Put the cup..."), the action is pushed to the stack as a continuation—a function awaiting its arguments. This allows the system to enter a disambiguation loop to resolve the destination before executing the final action.
Key Insights and Results
The integration of CPS and a modified PDA yields several advantages:
- Dialogue as Continuations: The system's internal logic mirrors human conversational patterns, effectively handling unresolved references and pauses.
- One-Shot Gesture Learning: Because actions are functions waiting for arguments, the system can instantly map a novel iconic gesture to a specific object or action.
- Complex Disambiguation: Qualitative results in VoxSim demonstrate the agent's ability to navigate multi-step loops, asking clarifying questions and processing multimodal feedback to resume original commands seamlessly.
Conclusion and Future Work
While the current system demonstrates high efficacy in a simulated environment, future work will focus on deployment on physical robotic platforms. This transition will require addressing the challenges of noisy sensor data and physical motor constraints not present in the current simulation.
Summary updated