A Unified Perspective on Multi-Domain and Multi-Task Learning

Yongxin Yang and Timothy HospedalesICLR 2015 · arXiv · OpenReview

Introduction

In machine learning, researchers often distinguish between Multi-Task Learning (MTL) (learning different problems in the same setting, like identifying different objects in one photo) and Multi-Domain Learning (MDL) (learning the same problem across different settings, like identifying the same object using different cameras).

This paper argues that the distinction between MTL and MDL is often artificial. Existing methods suffer from two major flaws:

  1. Atomic Assumptions: Most algorithms treat a task or domain as a simple, isolated ID (e.g., "Domain 1," "Domain 2"). This ignores rich metadata—like the fact that "Domain 1" might be a "Webcam" in a "Dark Room."
  2. Lack of Generalization: Because they treat domains as arbitrary IDs, these models cannot generalize to a new domain they haven't seen during training.

To address this, the authors introduce a unified neural network framework that treats MTL and MDL as two sides of the same coin. By using multivariate "semantic descriptors" (metadata) to describe tasks and domains, the framework improves performance on standard benchmarks and pioneers Zero-Shot Domain Adaptation (ZSDA)—the ability to synthesize a model for an entirely unseen environment without any training data.

Methodology: A Two-Sided Neural Network

The core of the paper is a novel two-sided neural network architecture designed to process both input features and semantic metadata simultaneously.

Two-sided neural network architecture
  • The Left Side ($f_P$): This branch performs representation learning on the standard input features ($x$).
  • The Right Side ($g_Q$): This branch performs "model construction" based on a semantic descriptor ($z$). This descriptor is a vector representing the metadata of the task or domain (e.g., a bitmask for "indoor/outdoor" or "microphone type").
  • The Interaction: The final prediction is an inner product of the outputs from both sides: $\hat{y} = f_P(x) \cdot g_Q(z)$.

A major theoretical contribution is the Unification Perspective. The authors mathematically demonstrate that several classic algorithms—including RMTL, FEDA, and GO-MTL—are actually special cases of this framework. These older methods simply make specific, often restrictive, assumptions about the structure of the descriptor matrix $Z$ and the weight matrices $P$ and $Q$.

Key Insights

  • Distributed Coding vs. 1-of-N: Instead of using a "one-hot" encoding for domains (where each domain is a unique, unrelated index), the authors use "distributed coding." If two domains share metadata (e.g., both use the same microphone), the model naturally shares parameters between them.
  • Zero-Shot Domain Adaptation (ZSDA): This is the "aha moment" of the paper. Just as Zero-Shot Learning allows a model to recognize a "Zebra" by knowing it has "stripes" (even if it's never seen a Zebra), ZSDA allows a model to function in a "Concert Hall" by knowing the hall's acoustic properties, even if it was only trained in "Studios" and "Living Rooms."
  • Simultaneous MDMT: The framework allows for concatenating task and domain descriptors, providing a straightforward way to handle complex problems where both the task and the environment vary simultaneously.

Experimental Results

The authors evaluate their framework across distinct experimental settings using four datasets: the School Dataset, an Audio Recognition task, Animals with Attributes (AwA), and a Restaurant & Consumer Dataset.

Performance comparison on the School Dataset
  • Superior Generalization: In the School Dataset, the proposed method achieved the lowest RMSE (9.37) compared to all baselines for Multi-Domain Learning.
  • ZSDA Success: In the Audio Recognition task, the framework significantly outperformed "Blind Transfer" (applying a general model to a new domain). For the most complex noise domain, the framework achieved an 18.75% error rate, while the next best baseline struggled at 59.38%.
  • MTL Boost: On the AwA dataset, using attribute descriptors for MTL improved multi-class accuracy by approximately 2% over standard Single-Task Learning (STL).
  • MDMT Performance: In the Restaurant dataset, the framework achieved an RMSE of 0.78, significantly outperforming the state-of-the-art GO-MTL (1.06).

Limitations and Future Work

While the framework is highly effective, the authors note a few areas for future exploration:

  • Discrete Descriptors: The current work focuses primarily on descriptors formed by discrete variables. Extending this to continuous and periodic variables (like time of day, brightness, or camera pose) is a necessary next step.
  • Observed Metadata: The framework assumes that the semantic descriptor is always available and observed. Dealing with missing or noisy metadata remains an open challenge.
  • Architecture Complexity: The paper uses a simple inner product for the interaction between the two sides of the network. Exploring more complex non-linear interactions could yield further improvements.

Summary updated

BibTeX
@inproceedings{hospedales2015unified,
  author = {Yongxin Yang and Timothy Hospedales},
  title = {{A Unified Perspective on Multi-Domain and Multi-Task Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2015},
  url = {http://arxiv.org/abs/1412.7489},
}
Figure