Transfer Learning in Conversational Analysis through Reusing Preprocessing Data as Supervisors

Joshua Yee Kim, Tongliang Liu, Kalina YacefarXiv 2021 · arXiv

Introduction

Analyzing human conversations for subjective metrics like emotion is a critical task for applications ranging from interview coaching to mental health monitoring. However, training deep learning models for these tasks is often hampered by the scarcity and noisiness of human-annotated labels. This paper introduces a novel Multi-Task Learning (MTL) approach that leverages pre-computed multimodal features—typically used only as inputs—as auxiliary supervisory signals to regularize the model and improve primary task performance.

Methods: The MONAH Framework and HAN-ROCK Architecture

The proposed system utilizes the MONAH framework to convert raw audio and video data into a text-based narrative. For example, a verbatim transcript like "The woman said no" is enriched with prosodic and visual cues to become "The woman raised chin. The woman sadly and slowly said no." This text, along with historical context, is fed into a Hierarchical Attention Network (HAN).

To integrate auxiliary tasks without degrading the primary task, the authors developed the HAN-ROCK architecture.

The HAN-ROCK architecture separates auxiliary and primary tasks into distinct branches, using a fusion module to prioritize the primary task's learning.

In this hierarchical structure, auxiliary tasks (such as predicting facial action units or vocal tone) are placed at a lower level. This ensures that while the primary task can benefit from the features learned by the auxiliary tasks, the auxiliary gradients do not negatively interfere with the primary task's specific representations. The authors define 16 auxiliary tasks across four categories: Facial Action Units, Prosody, Historical labels, and Future labels.

Key Insights: Capacity and Hierarchy

A critical finding of the research is the importance of asymmetric capacity. The authors discovered that the best results occur when the primary task branch is given maximum learning capacity (e.g., 256 GRUs), while the auxiliary branches are restricted to minimal capacity (e.g., 1 GRU). This "starvation" of the auxiliary branches forces them to learn only the most generalized, shared knowledge, acting as a powerful regularizer for the primary task rather than allowing the model to overfit on auxiliary noise.

Furthermore, the hierarchical placement of tasks prevents "negative transfer." By structuring the network so that auxiliary tasks act as foundational stepping stones, the primary task can build upon these representations without its specific learning objectives being overwritten by auxiliary gradients.

Experimental Results

The framework was evaluated on the IEMOCAP and SEMAINE datasets, covering both emotion classification and regression tasks (valence, activation, dominance).

Performance comparison on IEMOCAP and SEMAINE datasets showing improvements when adding auxiliary targets (ap, aphf) compared to the baseline.

As shown in the results, adding auxiliary targets (denoted as 'ap' for prosody/actions and 'aphf' for historical/future) significantly improved performance in 6 out of 8 primary tasks. Notably, on the IEMOCAP classification task, the weighted mean accuracy increased from a baseline of 0.625 to 0.715. The hierarchical HAN-ROCK architecture also proved superior to a traditional "flat" MTL approach in the majority of tests, confirming that structural prioritization is safer and more effective.

Conclusion

By treating preprocessed multimodal features as outputs rather than just inputs, this research demonstrates a cost-effective way to improve conversational analysis models. The combination of hierarchical task placement and asymmetric capacity allocation provides a robust framework for transfer learning that mitigates the risks of negative transfer and overfitting in data-scarce environments. Future work will explore automated methods for selecting the most effective auxiliary targets dynamically.

Summary updated

Abstract

Conversational analysis systems are trained using noisy human labels and often require heavy preprocessing during multi-modal feature extraction. Using noisy labels in single-task learning increases the risk of over-fitting. Auxiliary tasks could improve the performance of the primary task learning during the same training -- this approach sits in the intersection of transfer learning and multi-task learning (MTL). In this paper, we explore how the preprocessed data used for feature engineering can be re-used as auxiliary tasks, thereby promoting the productive use of data. Our main contributions are: (1) the identification of sixteen beneficially auxiliary tasks, (2) studying the method of distributing learning capacity between the primary and auxiliary tasks, and (3) studying the relative supervision hierarchy between the primary and auxiliary tasks. Extensive experiments on IEMOCAP and SEMAINE data validate the improvements over single-task approaches, and suggest that it may generalize across multiple primary tasks.
BibTeX
@misc{kim2021transfer,
  author = {Joshua Yee Kim and Tongliang Liu and Kalina Yacef},
  title = {{Transfer Learning in Conversational Analysis through Reusing Preprocessing Data as Supervisors}},
  year = {2021},
  eprint = {2112.03032},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.03032},
}
Figure