Learning Task-Preferred Inference Routes for Gradient De-Conflict in Multi-Output DNNs

Yi Sun, Xiaochang Hu, Xin Xu, Jian Li, Yifei Shi, Ling‐Li ZengIEEE TPAMI 2023 · arXiv

Introduction

Multi-output deep neural networks (MONs), such as multi-task and multi-exit architectures, are increasingly popular for their efficiency. However, they frequently encounter "gradient conflict," where different tasks pull shared parameters in opposing directions, hindering overall performance. This paper introduces DR-MGF (Dynamic Routes and Meta-weighted Gradient Fusion), a novel training paradigm designed to mitigate this interference by identifying task-specific "inference routes" and using meta-learning for gradient fusion.

The DR-MGF Framework

The authors propose a two-stage approach to handle gradient conflicts by recognizing that shared filters contribute differently to each task. Unlike methods that manually split a network, DR-MGF learns a "soft" disentanglement, maintaining the benefits of feature sharing while mitigating interference.

Overview of the DR-MGF framework, illustrating the disentanglement of network structure and meta-weighted gradient fusion.
  1. Disentanglement of Network Structure: This stage identifies task-preferred filters using learnable importance variables ($\nu$). By optimizing each task independently for a short period, the model learns "soft" inference routes without altering the physical architecture.
  2. Meta-weighted Gradient Fusion: Gradients are combined using a bi-level optimization strategy. An inner loop updates shared weights via a weighted fusion of task gradients, while an outer loop refines the importance variables ($\nu$) based on task performance on the objectives.

Experimental Evaluation

DR-MGF was tested across several standard benchmarks to prove its versatility:

  • CIFAR-10/100 & ImageNet: Evaluated on multi-exit classification tasks using Vgg-SDN, Resnet-SDN, and MSDnet.
  • NYUv2: A multi-task learning benchmark involving semantic segmentation, depth estimation, and surface normal prediction.

Results and Optimization Trajectories

DR-MGF consistently outperformed state-of-the-art methods like PCGrad, Cagrad, and Nash-MTL. For instance, on Resnet-SDN (CIFAR-100), it achieved a 61.84% average accuracy, a 1.52% improvement over the next best method. On NYUv2, it reached a performance degradation metric ($\Delta m$) of -8.39%, significantly better than baselines.

Optimization trajectories on a toy two-task problem, comparing DR-MGF with other gradient de-conflict methods.

As shown in the visualization of optimization trajectories, DR-MGF converges smoothly to the optimal solution for both tasks, whereas competing methods often oscillate or get stuck. This stability is partly attributed to the use of "expected gradients" calculated over an epoch, which prevents task-preferred routes from jittering during training.

Conclusion and Future Directions

While DR-MGF offers superior performance, it introduces computational overhead due to independent gradient calculations per task. Future work includes task grouping and threshold-based activation to improve efficiency. Additionally, the authors note that this disentanglement-and-fusion approach could be highly effective for alleviating "catastrophic forgetting" in continual learning scenarios.

Summary updated

Abstract

Multi-output deep neural networks (MONs) contain multiple output branches of various tasks, and these tasks typically share partial network filters, resulting in entangled inference routes between different tasks within the networks. Due to the divergent optimization objectives, the task gradients during training usually interfere with each other along the shared routes, which decreases the overall model performance. To address this issue, we propose a novel gradient de-conflict algorithm named DR-MGF (Dynamic Routes and Meta-weighted Gradient Fusion). Different from existing de-conflict methods, DR-MGF achieves gradient de-conflict in MONs by learning task-preferred inference routes. The proposed method is motivated by our experimental findings that the shared filters are not equally important for different tasks. By designing learnable task-specific importance variables, DR-MGF evaluates the importance of filters for different tasks. Through making the dominance of tasks over filters proportional to the task-specific importance of filters, DR-MGF can effectively reduce inter-task interference. These task-specific importance variables ultimately determine task-preferred inference routes at the end of training iterations. Extensive experimental results on CIFAR, ImageNet, and NYUv2 demonstrate that DR-MGF outperforms existing de-conflict methods. Furthermore, DR-MGF can be extended to general MONs without modifying the overall network structures.
BibTeX
@misc{sun2023learning,
  author = {Yi Sun and Xiaochang Hu and Xin Xu and Jian Li and Yifei Shi and Ling‐Li Zeng},
  title = {{Learning Task-Preferred Inference Routes for Gradient De-Conflict in Multi-Output DNNs}},
  year = {2023},
  eprint = {2305.19844},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.19844},
}
Figure