Multi-Task Learning via Time-Aware Neural ODE

Feiyang Ye, Xuehao Wang, Yu Zhang, Ivor W. Tsang

Introduction

Multi-Task Learning (MTL) aims to improve the performance of multiple related tasks by learning them simultaneously. The standard approach, Hard Parameter Sharing (HPS), utilizes a shared backbone to extract features followed by task-specific heads. However, HPS often suffers from "task competition," where gradients from different tasks conflict, leading to suboptimal performance. While existing methods attempt to balance losses or manipulate gradients, they rarely address the fundamental way task-specific features are derived from the shared representation.

The NORMAL Framework

The NORMAL (Neural Ordinal diffeRential equation based Multi-tAsk Learning) method proposes a paradigm shift: modeling task-specific transformations as a continuous evolution of features. Instead of discrete layers, NORMAL inserts a Time-Aware Neural ODE block between the shared feature extractor and the task-specific heads.

Comparison between standard Multi-Task Learning (top) and the proposed NORMAL framework (bottom).

At the heart of NORMAL is the concept of Task Positions ($p_i$). Each task $i$ is assigned a learnable "time" point along a continuous dynamic flow. To generate features for a specific task, the shared base feature $z(0)$ is evolved through the ODE until it reaches the task's unique position $p_i$. This allows the model to learn the optimal "depth" or transformation required for each task. To ensure stability and training efficiency, the authors employ a Heavy Ball Neural ODE (HBNODE), a second-order ODE that incorporates momentum-like dynamics.

Experimental Evaluation

The effectiveness of NORMAL was evaluated across four diverse MTL benchmarks:

  • Office-31 & Office-Home: Image classification across multiple domains.
  • NYUv2: A multi-modal vision dataset requiring semantic segmentation, depth estimation, and surface normal prediction.
  • CelebA: Large-scale facial attribute classification involving 40 tasks.

Results and Interpretability

NORMAL consistently outperformed state-of-the-art baselines, including specialized gradient manipulation techniques like PCGrad and Nash-MTL. For instance, on the Office-31 dataset, NORMAL achieved 94.80% average accuracy, surpassing the Equal Weighting baseline of 93.82%.

Beyond performance, NORMAL offers unique interpretability through its learned task positions. As shown in the evolution of $p_i$ during training, tasks that are semantically similar tend to converge toward similar positions in the continuous flow.

Evolution of learned task positions ($p_i$) over training epochs across different datasets.

This implicit task grouping suggests that the model naturally discovers relationships between tasks. In the CelebA dataset, attributes related to hair or facial structure often clustered together, validating that "time" in the ODE serves as a proxy for task specificity.

Conclusion

By reimagining MTL as a continuous dynamic flow, the NORMAL method provides a flexible and interpretable way to handle task-specific transformations. While it introduces some computational overhead due to the ODE solver, the gains in accuracy and the ability to visualize task relationships make it a promising direction for complex multi-task systems. Future work may involve extending this "task position" concept to other architectures and domains like Natural Language Processing.

Summary updated

Abstract

Multi-Task Learning (MTL) is a well-established paradigm for learning shared models for a diverse set of tasks. Moreover, MTL improves data efficiency by jointly training all tasks simultaneously. However, directly optimizing the losses of all the tasks may lead to imbalanced performance on all the tasks due to the competition among tasks for the shared parameters in MTL models. Many MTL methods try to mitigate this problem by dynamically weighting task losses or manipulating task gradients. Different from existing studies, in this paper, we propose a Neural Ordinal diffeRential equation based Multi-tAsk Learning (NORMAL) method to alleviate this issue by modeling task-specific feature transformations from the perspective of dynamic flows built on the Neural Ordinary Differential Equation (NODE). Specifically, the proposed NORMAL model designs a time-aware neural ODE block to learn task-specific time information, which determines task positions of feature transformations in the dynamic flow, in NODE automatically via gradient descent methods. In this way, the proposed NORMAL model handles the problem of competing shared parameters by learning task positions. Moreover, the learned task positions can be used to measure the relevance among different tasks. Extensive experiments show that the proposed NORMAL model outperforms state-of-the-art MTL models.
BibTeX
@inproceedings{ye2023multi,
  author = {Feiyang Ye and Xuehao Wang and Yu Zhang and Ivor W. Tsang},
  title = {{Multi-Task Learning via Time-Aware Neural ODE}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/500},
}
Figure