Episodic Multi-Task Learning with Heterogeneous Neural Processes

Jiayi Shen, Xiantong Zhen, Cheems Wang, Marcel Worring

One of the most persistent hurdles in machine learning is data insufficiency. While multi-task learning (MTL) and meta-learning are standard tools for handling small datasets, they are rarely combined effectively. To bridge this gap, researchers have introduced Heterogeneous Neural Processes (HNPs), a novel approach that allows models to learn from both past experiences and the relationships between different tasks within a single training "episode."

By combining episodic meta-learning with multi-task learning through a hierarchical Bayesian framework and Transformer-structured inference, HNPs can simultaneously reuse knowledge from previous episodes and exploit correlations between different tasks to make accurate predictions from just a few examples.

The Data Insufficiency Problem

In many real-world scenarios, massive labeled datasets are unavailable. To combat this, researchers typically turn to two paradigms:

  1. Meta-Learning: Learning how to learn by extracting "meta-knowledge" from past tasks to adapt quickly to a new one.
  2. Multi-Task Learning (MTL): Learning multiple related tasks at once to exploit their underlying correlations.
Comparison of Episodic Single-Task and Multi-Task Learning setups.

However, standard meta-learning usually focuses on a single task per episode (episodic single-task learning). Conversely, standard MTL models often "forget" what they learned in previous training sessions.

Furthermore, there is an expressiveness bottleneck in existing Neural Processes (NPs). Vanilla NPs use a single global latent variable to represent a task, which is not powerful enough to capture the nuances of multiple heterogeneous (different but related) tasks occurring simultaneously in a single episode.

Heterogeneous Neural Processes (HNPs)

The core contribution of this work is the HNP, which evolves the Neural Process family by introducing a hierarchical functional space:

  • Hierarchical Bayes Framework: Instead of relying on a single latent variable, HNPs use a two-level hierarchy. A global latent representation ($z$) captures task-specific context, while a set of local latent parameters ($w$) acts as a prediction-aware decoder tailored to each specific task.
  • Transformer-Structured Inference: The inference modules are based on the Transformer architecture. They utilize Multi-Headed Self-Attention (MSA) to fuse context samples with learnable meta-knowledge tokens. This mechanism allows the model to "reason" about how current data relates to past experiences.
  • Variational Inference: The model is optimized using an approximate Evidence Lower Bound (ELBO). This ensures that the latent variables inferred from limited "context" data remain close to the true distributions observed during training.

Key Insights

  • Episodic Multi-Tasking is Key: By training on multiple related tasks in every episode, the model learns to generalize the ability to explore task-relatedness. This makes it significantly more robust when encountering a brand-new set of tasks at test time.
  • Decoupling Representation and Parameters: Treating the local latent variables ($w$) as parameters for a decoder—rather than just additional features—gives the model the flexibility needed to handle diverse, heterogeneous tasks.
  • Meta-Knowledge as Tokens: Using learnable tokens within a Transformer allows the model to maintain a "memory" of inductive biases across tasks and episodes in a unified, computationally efficient manner.
  • Uncertainty Matters: The probabilistic nature of HNPs allows them to quantify uncertainty effectively. In regression tasks, HNPs provide much more reliable "confidence intervals" (showing lower variance near known data points) than standard NPs or Attentive NPs.

Experimental Results

The authors evaluated HNPs on both regression and classification benchmarks, including a 1D Regression toy dataset, Office-Home (~15,500 images, 65 categories), and DomainNet (~0.6 million images, 345 categories).

Classification results on Office-Home and DomainNet datasets.
  • Superior Accuracy: In the Office-Home 1-shot classification task (4 tasks, 5-way), HNPs achieved a remarkable 76.29% accuracy. This significantly outperforms established baselines like MAML (60.58%), Prototypical Networks (57.19%), and vanilla NPs (33.66%).
  • Smoothness in Regression: HNPs produced significantly smoother and more accurate predictive curves in 1D regression compared to NPs and ANPs, which often struggled with "jagged" predictions when data was sparse.
  • Ablation Success: Removing either the global or local latent variables led to a sharp drop in performance (a ~3-5% decrease in accuracy), proving that the hierarchical structure is vital for capturing complex task relationships.

Limitations and Future Work

While highly effective, HNPs currently require all tasks within a single episode to share the same target space (e.g., the same set of possible labels). This may limit applicability in scenarios where tasks are wildly different. Additionally, the hierarchical sampling and Transformer layers introduce computational overhead; inference takes roughly 0.15s per iteration, compared to 0.05s for vanilla NPs.

Future directions include extending the framework to handle tasks with different target spaces by developing "higher-order task-relatedness" modules to further improve knowledge sharing across diverse domains.

Summary updated

Abstract

This paper focuses on the data-insufficiency problem in multi-task learning within an episodic training setup. Specifically, we explore the potential of heterogeneous information across tasks and meta-knowledge among episodes to effectively tackle each task with limited data. Existing meta-learning methods often fail to take advantage of crucial heterogeneous information in a single episode, while multi-task learning models neglect reusing experience from earlier episodes. To address the problem of insufficient data, we develop Heterogeneous Neural Processes (HNPs) for the episodic multi-task setup. Within the framework of hierarchical Bayes, HNPs effectively capitalize on prior experiences as meta-knowledge and capture task-relatedness among heterogeneous tasks, mitigating data-insufficiency. Meanwhile, transformer-structured inference modules are designed to enable efficient inferences toward meta-knowledge and task-relatedness. In this way, HNPs can learn more powerful functional priors for adapting to novel heterogeneous tasks in each meta-test episode. Experimental results show the superior performance of the proposed HNPs over typical baselines, and ablation studies verify the effectiveness of the designed inference modules.
BibTeX
@inproceedings{shen2023episodic,
  author = {Jiayi Shen and Xiantong Zhen and Cheems Wang and Marcel Worring},
  title = {{Episodic Multi-Task Learning with Heterogeneous Neural Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=FXU4aR2uif},
}
Figure