Episodic Multi-Task Learning with Heterogeneous Neural Processes

Jiayi Shen, Xiantong Zhen, Qi Wang, Marcel Worring

Abstract

This paper focuses on the data-insufficiency problem in multi-task learning within an episodic training setup. Specifically, we explore the potential of heterogeneous information across tasks and meta-knowledge among episodes to effectively tackle each task with limited data. Existing meta-learning methods often fail to take advantage of crucial heterogeneous information in a single episode, while multi-task learning models neglect reusing experience from earlier episodes. To address the problem of insufficient data, we develop Heterogeneous Neural Processes (HNPs) for the episodic multi-task setup. Within the framework of hierarchical Bayes, HNPs effectively capitalize on prior experiences as meta-knowledge and capture task-relatedness among heterogeneous tasks, mitigating data-insufficiency. Meanwhile, transformer-structured inference modules are designed to enable efficient inferences toward meta-knowledge and task-relatedness. In this way, HNPs can learn more powerful functional priors for adapting to novel heterogeneous tasks in each meta-test episode. Experimental results show the superior performance of the proposed HNPs over typical baselines, and ablation studies verify the effectiveness of the designed inference modules.

Introduction: The Data-Insufficiency Gap

Deep learning models typically require vast amounts of labeled data to perform well. However, in domains like medical imaging or personalized robotics, data is often scarce. While Meta-Learning (learning to learn) enables rapid adaptation to new tasks, it typically focuses on a single task per episode. Conversely, Multi-Task Learning (MTL) leverages correlations between multiple tasks but usually requires large datasets and lacks a mechanism to reuse experience across training episodes.

The authors identify a critical gap: Episodic Multi-Task Learning. Standard models, such as vanilla Neural Processes (NPs), face an "expressiveness bottleneck" because they rely on a single global latent variable. This representation is often insufficient for the complex, heterogeneous information present when multiple distinct tasks are bundled into a single learning episode.

Comparison of Episodic Single-Task Learning and the proposed Episodic Multi-Task Learning framework.

Methodology: Heterogeneous Neural Processes (HNPs)

To overcome these limitations, the authors propose Heterogeneous Neural Processes (HNPs). The framework introduces two primary innovations:

  1. Hierarchical Bayesian Framework: HNPs replace the single latent variable with a hierarchy. Global Latent Representations ($z^m_\tau$) encode task-specific context, while Local Latent Parameters ($w^m_{\tau, 1:O}$) function as prediction-aware decoders. By generating its own decoder weights based on observed context, the model can represent much more complex functional priors.
  2. Transformer-Structured Inference: The model employs attention mechanisms to fuse context samples with learnable meta-knowledge tokens ($\omega$ and $\nu$). This allows the HNP to reason about task-relatedness within an episode and leverage past experiences, similar to how large language models utilize context.

Experimental Evaluation

The HNP framework was evaluated across several benchmarks, including 1D regression for uncertainty quantification and large-scale image classification datasets like Office-Home (65 categories across four domains) and DomainNet (six domains and 345 categories).

Key Results

HNPs consistently outperformed state-of-the-art meta-learning and multi-task learning baselines. On the Office-Home benchmark (4-task, 5-way, 1-shot), HNPs achieved an accuracy of 76.29%, significantly higher than MAML (60.58%) and Prototypical Networks (57.19%).

Performance comparison of HNPs against various baselines on the Office-Home and DomainNet datasets.

Beyond accuracy, HNPs demonstrated superior uncertainty quantification in 1D regression tasks, producing smoother predictive curves and lower Negative Log-Likelihood (NLL) than Attentive Neural Processes (ANPs). In real-world applications, the HNP-based approach surpassed standard U-Net baselines for brain tumor segmentation, proving its versatility.

Conclusion and Future Directions

HNPs effectively bridge the gap between meta-learning and multi-task learning by harvesting "transferable information" across heterogeneous tasks. While the transformer-based inference introduces a slight computational overhead (0.15s vs. 0.05s per iteration for standard NPs), the performance gains are substantial. Future work aims to extend HNPs to scenarios where tasks do not share the same target space, requiring even more sophisticated models of task-relatedness.

Summary updated

Abstract

This paper focuses on the data-insufficiency problem in multi-task learning within an episodic training setup. Specifically, we explore the potential of heterogeneous information across tasks and meta-knowledge among episodes to effectively tackle each task with limited data. Existing meta-learning methods often fail to take advantage of crucial heterogeneous information in a single episode, while multi-task learning models neglect reusing experience from earlier episodes. To address the problem of insufficient data, we develop Heterogeneous Neural Processes (HNPs) for the episodic multi-task setup. Within the framework of hierarchical Bayes, HNPs effectively capitalize on prior experiences as meta-knowledge and capture task-relatedness among heterogeneous tasks, mitigating data-insufficiency. Meanwhile, transformer-structured inference modules are designed to enable efficient inferences toward meta-knowledge and task-relatedness. In this way, HNPs can learn more powerful functional priors for adapting to novel heterogeneous tasks in each meta-test episode. Experimental results show the superior performance of the proposed HNPs over typical baselines, and ablation studies verify the effectiveness of the designed inference modules.
BibTeX
@inproceedings{shen2023episodic,
  author = {Jiayi Shen and Xiantong Zhen and Qi Wang and Marcel Worring},
  title = {{Episodic Multi-Task Learning with Heterogeneous Neural Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=FXU4aR2uif},
}
Figure