Introduction
Analyzing human conversations for subjective metrics like emotion is a critical task for applications ranging from interview coaching to mental health monitoring. However, training deep learning models for these tasks is often hampered by the scarcity and noisiness of human-annotated labels. This paper introduces a novel Multi-Task Learning (MTL) approach that leverages pre-computed multimodal features—typically used only as inputs—as auxiliary supervisory signals to regularize the model and improve primary task performance.
Methods: The MONAH Framework and HAN-ROCK Architecture
The proposed system utilizes the MONAH framework to convert raw audio and video data into a text-based narrative. For example, a verbatim transcript like "The woman said no" is enriched with prosodic and visual cues to become "The woman raised chin. The woman sadly and slowly said no." This text, along with historical context, is fed into a Hierarchical Attention Network (HAN).
To integrate auxiliary tasks without degrading the primary task, the authors developed the HAN-ROCK architecture.
In this hierarchical structure, auxiliary tasks (such as predicting facial action units or vocal tone) are placed at a lower level. This ensures that while the primary task can benefit from the features learned by the auxiliary tasks, the auxiliary gradients do not negatively interfere with the primary task's specific representations. The authors define 16 auxiliary tasks across four categories: Facial Action Units, Prosody, Historical labels, and Future labels.
Key Insights: Capacity and Hierarchy
A critical finding of the research is the importance of asymmetric capacity. The authors discovered that the best results occur when the primary task branch is given maximum learning capacity (e.g., 256 GRUs), while the auxiliary branches are restricted to minimal capacity (e.g., 1 GRU). This "starvation" of the auxiliary branches forces them to learn only the most generalized, shared knowledge, acting as a powerful regularizer for the primary task rather than allowing the model to overfit on auxiliary noise.
Furthermore, the hierarchical placement of tasks prevents "negative transfer." By structuring the network so that auxiliary tasks act as foundational stepping stones, the primary task can build upon these representations without its specific learning objectives being overwritten by auxiliary gradients.
Experimental Results
The framework was evaluated on the IEMOCAP and SEMAINE datasets, covering both emotion classification and regression tasks (valence, activation, dominance).
As shown in the results, adding auxiliary targets (denoted as 'ap' for prosody/actions and 'aphf' for historical/future) significantly improved performance in 6 out of 8 primary tasks. Notably, on the IEMOCAP classification task, the weighted mean accuracy increased from a baseline of 0.625 to 0.715. The hierarchical HAN-ROCK architecture also proved superior to a traditional "flat" MTL approach in the majority of tests, confirming that structural prioritization is safer and more effective.
Conclusion
By treating preprocessed multimodal features as outputs rather than just inputs, this research demonstrates a cost-effective way to improve conversational analysis models. The combination of hierarchical task placement and asymmetric capacity allocation provides a robust framework for transfer learning that mitigates the risks of negative transfer and overfitting in data-scarce environments. Future work will explore automated methods for selecting the most effective auxiliary targets dynamically.
Summary updated