Gated Multimodal Fusion with Contrastive Learning for Turn-Taking Prediction in Human-Robot Dialogue

Jiudong Yang, Peiying Wang, Yi Zhu, Mingchao Feng, Meng Chen, Xiaodong He

Introduction

Effective turn-taking is a cornerstone of natural human-robot interaction (HRI). For a robot to be perceived as a competent conversational partner, it must accurately predict when a user has finished speaking (turn-switching) and when they are merely pausing to catch their breath or collect their thoughts (turn-holding). Errors in this process lead to "false barge-ins"—where the robot interrupts the user—or awkward, prolonged silences that disrupt the flow of dialogue.

Despite its importance, turn-taking prediction remains a significant challenge due to three primary factors: the scarcity of large-scale multimodal datasets, the difficulty of effectively fusing disparate data types (text, audio, and timing), and the inherent "class imbalance" where turn-holding events are significantly rarer than turn-switching events.

The Gated Multimodal Fusion (GMF) Framework

To address these challenges, researchers from JD AI and the University of Cambridge introduced the Gated Multimodal Fusion (GMF) model. This architecture is designed to process and integrate three distinct streams of conversational data:

  1. Semantic Features: A Transformer-based encoder processes the current utterance's text alongside the preceding dialogue context. This ensures the model understands the linguistic meaning and conversational history.
  2. Acoustic Features: An 18-layer ResNet analyzes the final two seconds of the audio signal, extracting prosodic cues such as pitch, energy, and spectral characteristics.
  3. Timing Features: A Multi-Layer Perceptron (MLP) processes metadata, including the duration of the current speech segment and the user's speaking rate.

Gated Fusion Mechanism

Unlike traditional models that simply concatenate feature vectors, the GMF model employs a gating mechanism. This gate dynamically calculates weights for each modality, allowing the model to prioritize the most relevant information at any given moment. For instance, if the audio signal is noisy, the model can rely more heavily on semantic cues to determine if a sentence is grammatically complete.

Overcoming Class Imbalance with Contrastive Learning

A major contribution of this work is the application of self-supervised Contrastive Learning (CL) to solve the class imbalance problem. In real-world dialogue, "hold" instances are infrequent, making it difficult for models to learn the subtle differences between a pause and an end-of-turn.

The researchers generated synthetic negative examples by "corrupting" successful turns—for example, by truncating a finished sentence to simulate an unfinished "hold." During training, the CL strategy encourages the model to pull similar conversational representations together in the embedding space while pushing dissimilar ones (like a true end-of-turn vs. a synthetic hold) apart. This approach significantly enhances the model's sensitivity to the nuances of turn-holding.

Experimental Setup and Dataset

The model was evaluated using a new, large-scale corpus consisting of over 5,000 real-world human-robot dialogues. These interactions were sourced from a commercial Interactive Voice Response (IVR) system used for delivery appointments. The dataset includes both raw audio and manual transcriptions, with labels for:

  • Endpointing: Determining if the user has finished their turn.
  • Barge-in: Distinguishing between meaningful user interruptions and background noise.

Results and Discussion

The GMF model with Contrastive Learning (GMF w/ CL) established a new state-of-the-art benchmark for turn-taking prediction.

  • Endpointing Performance: The model achieved a Macro-F1 score of 0.761, representing an 11.8% absolute improvement over the previous state-of-the-art Mixture of Experts (MoE) model.
  • Barge-in Detection: The model reached a Macro-F1 score of 0.826, significantly outperforming baseline LSTM and MoE architectures.

Ablation studies confirmed the necessity of the multimodal approach. Removing any single modality—semantic, acoustic, or timing—resulted in a measurable decrease in performance. Furthermore, the inclusion of dialogue context proved critical; without the history of the conversation, the model's ability to predict turn-taking dropped significantly, highlighting that turn-taking is a contextual, rather than purely local, phenomenon.

Conclusion

The Gated Multimodal Fusion model represents a significant step forward in making human-robot dialogue feel more fluid and natural. By combining sophisticated gating mechanisms with self-supervised contrastive learning, the researchers have mitigated the long-standing issues of data imbalance and ineffective feature fusion.

Future research will focus on expanding the model to include visual modalities, such as eye-gaze and gestures, and optimizing the computational efficiency of the Transformer and ResNet encoders to ensure low-latency performance in real-time robotic applications.

Summary updated

BibTeX
@inproceedings{yang2022gated,
  author = {Jiudong Yang and Peiying Wang and Yi Zhu and Mingchao Feng and Meng Chen and Xiaodong He},
  title = {{Gated Multimodal Fusion with Contrastive Learning for Turn-Taking Prediction in Human-Robot Dialogue}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747056},
}
Figure