Zero-Shot Task Transfer

Arghya Pal, Vineeth N. Balasubramanian

Abstract

In this work, we present a novel meta-learning algorithm that regresses model parameters for novel tasks for which no ground truth is available (zero-shot tasks). In order to adapt to novel zero-shot tasks, our meta-learner learns from the model parameters of known tasks (with ground truth) and the correlation of known tasks to zero-shot tasks. Such intuition finds its foothold in cognitive science, where a subject (human baby) can adapt to a novel concept (depth understanding) by correlating it with old concepts (hand movement or self-motion), without receiving an explicit supervision. We evaluated our model on the Taskonomy dataset, with four tasks as zero-shot: surface normal, room layout, depth and camera pose estimation. These tasks were chosen based on the data acquisition complexity and the complexity associated with the learning process using a deep network. Our proposed methodolgy outperforms state-of-the-art models (which use ground truth) on each of our zero-shot tasks, showing promise on zero-shot task transfer. We also conducted extensive experiments to study the various choices of our methodology, as well as showed how the proposed method can also be used in transfer learning. To the best of our knowledge, this is the first such effort on zero-shot learning in the task space.

Introduction

Deep learning is notoriously data-hungry. While massive datasets exist for common tasks like object detection, many specialized vision tasks—such as estimating surface normals, room layouts, or 3D camera poses—require expensive sensors, expert annotation, or grueling manual labor.

Standard "Zero-Shot Learning" usually refers to recognizing new object categories. This paper tackles a more ambitious problem: Zero-Shot Task Transfer. The goal is to obtain a functional model for a brand-new task (like depth estimation) by only using the model parameters of other known tasks (like edge detection or colorization) and a map of how these tasks relate to one another.

Conceptual Framework: The Meta-Manifold

The core idea is that the parameters of different vision tasks lie on a structured "meta-manifold." If you know where several tasks sit on this manifold, you can predict where a new, related task should be located. This shifts the focus from learning from data points to learning from the relationships between task solutions themselves.

Conceptual overview of the Data Manifold versus the Meta Manifold of task parameters.

Methodology: TTNet

The authors propose a meta-learning algorithm called TTNet. The framework treats network weights as points in a high-dimensional space that can be modeled, interpolated, and extrapolated.

1. Task Representation and Correlation

Every task is assumed to use an Encoder-Decoder architecture. To guide the meta-learner, the authors use "wisdom-of-the-crowd" task correlations ($\Gamma$). They asked 30 annotators to rate the correlation between pairs of tasks (e.g., how much does knowing "edges" help with "surface normals"?) to provide a weak supervision signal for navigating the parameter space.

2. The Meta-Learner Architecture

The meta-learner network takes the parameters of $m$ known tasks and the correlation matrix as input. It then regresses (predicts) the encoder and decoder parameters for the target zero-shot task ($m+1$).

The TTNet architecture for zero-shot task transfer, predicting parameters for task m+1 based on m known tasks.

3. Training and Consistency

TTNet is trained using a dual-mode approach. In Self Mode, it learns to reconstruct a known task's parameters from itself. In Transfer Mode, it learns to predict a known task's parameters using only the parameters of other tasks. Crucially, a Data Model Consistency Loss ensures that the predicted weights, when applied to an actual image, produce a result that is consistent with the original data manifold.

Key Results

Remarkably, TTNet often outperformed state-of-the-art models that were actually trained with ground truth data:

  • Surface Normal Estimation: On the Taskonomy dataset, TTNet achieved a Mean Error of 15.10, outperforming supervised models like SkipNet (20.21) and GeoNet (19.00).
  • Room Layout: TTNet achieved a pixel error of 7.10, beating the supervised Taskonomy baseline of 8.0.
  • Generalization: When tested on the outdoor Cityscapes dataset for surface normals, the zero-shot model (trained only on indoor data) captured finer details (like trees and humans) than existing supervised models.

Conclusion

TTNet demonstrates that human intuition about task relationships is a powerful enough signal to help a meta-learner navigate the parameter space. By treating parameters as data, this work provides a viable path for obtaining high-performance models for specialized tasks where ground truth data is impossible or too expensive to acquire.

Summary updated

Abstract

In this work, we present a novel meta-learning algorithm that regresses model parameters for novel tasks for which no ground truth is available (zero-shot tasks). In order to adapt to novel zero-shot tasks, our meta-learner learns from the model parameters of known tasks (with ground truth) and the correlation of known tasks to zero-shot tasks. Such intuition finds its foothold in cognitive science, where a subject (human baby) can adapt to a novel concept (depth understanding) by correlating it with old concepts (hand movement or self-motion), without receiving an explicit supervision. We evaluated our model on the Taskonomy dataset, with four tasks as zero-shot: surface normal, room layout, depth and camera pose estimation. These tasks were chosen based on the data acquisition complexity and the complexity associated with the learning process using a deep network. Our proposed methodolgy outperforms state-of-the-art models (which use ground truth) on each of our zero-shot tasks, showing promise on zero-shot task transfer. We also conducted extensive experiments to study the various choices of our methodology, as well as showed how the proposed method can also be used in transfer learning. To the best of our knowledge, this is the first such effort on zero-shot learning in the task space.
BibTeX
@inproceedings{pal2019zero,
  author = {Arghya Pal and Vineeth N. Balasubramanian},
  title = {{Zero-Shot Task Transfer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=544LvhHq9PSv},
}
Figure