Introduction
Multi-output deep neural networks (MONs), such as multi-task and multi-exit architectures, are increasingly popular for their efficiency. However, they frequently encounter "gradient conflict," where different tasks pull shared parameters in opposing directions, hindering overall performance. This paper introduces DR-MGF (Dynamic Routes and Meta-weighted Gradient Fusion), a novel training paradigm designed to mitigate this interference by identifying task-specific "inference routes" and using meta-learning for gradient fusion.
The DR-MGF Framework
The authors propose a two-stage approach to handle gradient conflicts by recognizing that shared filters contribute differently to each task. Unlike methods that manually split a network, DR-MGF learns a "soft" disentanglement, maintaining the benefits of feature sharing while mitigating interference.
- Disentanglement of Network Structure: This stage identifies task-preferred filters using learnable importance variables ($\nu$). By optimizing each task independently for a short period, the model learns "soft" inference routes without altering the physical architecture.
- Meta-weighted Gradient Fusion: Gradients are combined using a bi-level optimization strategy. An inner loop updates shared weights via a weighted fusion of task gradients, while an outer loop refines the importance variables ($\nu$) based on task performance on the objectives.
Experimental Evaluation
DR-MGF was tested across several standard benchmarks to prove its versatility:
- CIFAR-10/100 & ImageNet: Evaluated on multi-exit classification tasks using Vgg-SDN, Resnet-SDN, and MSDnet.
- NYUv2: A multi-task learning benchmark involving semantic segmentation, depth estimation, and surface normal prediction.
Results and Optimization Trajectories
DR-MGF consistently outperformed state-of-the-art methods like PCGrad, Cagrad, and Nash-MTL. For instance, on Resnet-SDN (CIFAR-100), it achieved a 61.84% average accuracy, a 1.52% improvement over the next best method. On NYUv2, it reached a performance degradation metric ($\Delta m$) of -8.39%, significantly better than baselines.
As shown in the visualization of optimization trajectories, DR-MGF converges smoothly to the optimal solution for both tasks, whereas competing methods often oscillate or get stuck. This stability is partly attributed to the use of "expected gradients" calculated over an epoch, which prevents task-preferred routes from jittering during training.
Conclusion and Future Directions
While DR-MGF offers superior performance, it introduces computational overhead due to independent gradient calculations per task. Future work includes task grouping and threshold-based activation to improve efficiency. Additionally, the authors note that this disentanglement-and-fusion approach could be highly effective for alleviating "catastrophic forgetting" in continual learning scenarios.
Summary updated