Task Selection and Assignment for Multi-modal Multi-task Dialogue Act Classification with Non-stationary Multi-armed Bandits

Xiangheng He, Junjie Chen, Björn W. Schuller

Abstract

Multi-task learning (MTL) is a powerful paradigm for improving model performance on a primary task by leveraging shared representations from auxiliary tasks. However, static or random task selection often leads to "negative transfer," where irrelevant auxiliary tasks degrade performance. This paper introduces a dynamic training strategy for Dialogue Act Classification (DAC) using non-stationary multi-armed bandits (MAB). By employing discounted Thompson Sampling, the proposed MAB Controller adaptively selects the most beneficial auxiliary tasks throughout the training process. Experimental results on the IEMOCAP dataset demonstrate that this approach significantly outperforms standard MTL baselines and state-of-the-art models, particularly in maintaining high performance on minority classes.

Introduction

Dialogue Act Classification (DAC) is a critical component in conversational AI, requiring the interpretation of speaker intentions from multi-modal inputs like text and audio. While multi-task learning (MTL) offers a way to enrich these representations using auxiliary tasks (e.g., emotion or speaker recognition), it faces two significant challenges:

  1. Task Selection: Identifying which auxiliary tasks are truly beneficial.
  2. Task Assignment: Determining when a specific task should be trained, as its utility may fluctuate during the learning process.

Static MTL approaches often fail to account for these dynamics, sometimes resulting in performance worse than single-task models. This research addresses these issues by framing task selection as a non-stationary multi-armed bandit problem.

Methodology

The proposed architecture consists of a multi-modal Transformer guided by an MAB Controller.

Model Architecture

The core model is a Transformer encoder-decoder. It processes textual features from BERT and audio features from VGGish, fusing them via multi-modal self-attention. The decoder then generates predictions for the selected task.

The proposed architecture featuring the MAB Controller overseeing task selection for the multi-modal Transformer.

MAB Controller

The selection process is modeled as a Multi-Armed Bandit problem where each "arm" represents a task. To handle the non-stationary nature of task utility—where a task's helpfulness changes as the model learns—the authors utilize discounted Thompson Sampling (TS) with Gaussian priors.

The "reward" for the bandit is defined as the improvement in validation performance on the primary task. A discount factor ($\gamma$) is applied to prioritize recent performance gains, allowing the controller to adapt to the model's evolving needs.

Experimental Setup

The study utilizes an extended version of the IEMOCAP dataset. The primary task is DAC (8-class and 12-class), supported by auxiliary tasks including emotion classification, speaker classification, and regression for sentiment dimensions (Arousal, Valence, and Dominance). The dataset is split into 70% training, 20% validation, and 10% testing.

Results and Discussion

The MAB approach demonstrated superior performance across all evaluated metrics compared to both single-task (ST) and random multi-task (MT) baselines.

Dynamic Task Utility

A key finding is the dynamic shift in task preference over time. As shown in the selection probability visualization, the model initially focuses on the primary task (DA) before exploring auxiliary tasks. In later stages, it converges on specific beneficial tasks like emotion classification and sentiment regression, while discarding less useful ones like speaker classification.

Visualization of task selection probability over training rounds, showing the transition from exploration to exploitation of specific auxiliary tasks.

Performance Comparison

The proposed method achieved significant gains, particularly on the 12-class dataset where it reached an F1 score of 0.6858, surpassing the previous state-of-the-art.

Metric Single-Task (ST) Multi-Task (Random) Proposed (MAB)
UAR 0.618 0.522 0.667
F1 Score 0.604 0.499 0.655
Accuracy 0.675 0.608 0.683

Notably, the random MTL baseline performed worse than the single-task model, highlighting the prevalence of negative transfer in unguided multi-task settings. Furthermore, the MAB strategy provided much-needed stability for minority dialogue acts, preventing the F1 scores from dropping to zero as seen in baseline models.

Conclusion

This work demonstrates that non-stationary multi-armed bandits can effectively manage task selection in multi-modal MTL. By adaptively assigning tasks, the model avoids negative transfer and improves overall robustness. Future research will explore the scalability of this approach to larger sets of auxiliary tasks and its applicability to other domains such as computer vision.

Summary updated

BibTeX
@inproceedings{he2024task,
  author = {Xiangheng He and Junjie Chen and Björn W. Schuller},
  title = {{Task Selection and Assignment for Multi-modal Multi-task Dialogue Act Classification with Non-stationary Multi-armed Bandits}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447862},
}
Figure