Task Understanding from Confusing Multi-task Data

Xin Su, Yizhou Jiang, Shangqi Guo, Feng ChenICML 2020 · OpenReview

TL;DR

Existing multi-task learning (MTL) models rely heavily on manual task labels to know which data point belongs to which objective. This paper introduces Confusing Supervised Learning (CSL), a framework that allows a machine to autonomously "de-confuse" mixed datasets and learn multiple distinct task mappings simultaneously—without any manual task annotations.

What Problem Does This Paper Address?

In traditional supervised learning, we assume a single mapping from an input $X$ to an output $Y$. In multi-task learning, we acknowledge that the same input $X$ might have different valid outputs depending on the goal. For example, an image of a red apple could be labeled "Red" for a color-identification task or "Apple" for an object-recognition task.

Comparison of traditional multi-task learning, which requires manual task assignment, and the proposed approach that autonomously de-confuses data.

The "Confusing Mapping Challenge" occurs when these diverse samples are thrown into one big bucket without labels indicating which task is which. A standard neural network trained on this mixed data would likely just learn the "mean" of all tasks, failing to perform accurately on any specific one.

Graphical models comparing single-task, multi-task, and Confusing Supervised Learning (CSL), illustrating how standard models fail by learning the 'mean' of confusing data.

The authors argue that true artificial intelligence should be able to look at this confusing data and infer the underlying task concepts on its own, much like a human can distinguish between different contexts without explicit instructions.

Methods: The CSL-Net Framework

To solve this, the authors propose CSL-Net, a novel architecture that consists of two primary components working in tandem:

  1. Deconfusing-Net ($h$): This network acts as a "traffic controller." It takes an input-output pair $(x, y)$ and predicts which task it belongs to, outputting a one-hot vector.
  2. Mapping-Net ($g$): A multi-branch network where each branch represents a specific task mapping.

The Iterative Deconfusing Algorithm

Because the Deconfusing-Net needs to output a discrete choice (a one-hot vector assigning a sample to a task), standard gradient descent cannot be applied directly. To overcome this, the authors use an iterative, two-step training approach:

  • Step A: Fix the Deconfusing-Net's parameters. Train the Mapping-Net branches using the samples currently assigned to them by the Deconfusing-Net.
  • Step B: Fix the Mapping-Net's parameters. Update the Deconfusing-Net by checking which Mapping-Net branch produces the smallest error for each sample. The algorithm treats that lowest-error branch as the "temporary ground truth" for the Deconfusing-Net to learn.

To avoid getting stuck in poor local minima during this iterative process, the authors utilize a few-shot warm-up. By providing a tiny amount of labeled data (e.g., just 5 samples), the network gets a "hint" of the correct task directions before fully autonomous training begins.

Key Insights and Ideas

  • Task Discovery via Error Minimization: The core "aha" moment of CSL is that a system can discover tasks simply by trying to minimize total error. If a single model cannot fit the data, but two separate models can fit subsets of the data perfectly, the system naturally "clusters" the data into distinct tasks.
  • Structural Risk Minimization: The authors provide a theoretical proof showing that the optimal number of tasks can be determined by finding the minimum number of branches that brings the training risk close to zero.
  • Human-like Understanding: By separating the "what is the task?" question from "how do I do the task?", the model mimics human cognitive processes for handling ambiguous information.

Datasets and Key Results

The researchers evaluated their approach across several benchmarks, ranging from synthetic to real-world data:

  • Function Regression: On synthetic datasets featuring overlapping mathematical functions, traditional supervised learning produced a "mean" line that fit none of the functions. CSL-Net successfully separated and learned 2-3 distinct functions with near-zero loss.
  • Colorful-MNIST: An extension of the classic MNIST dataset where digits are colored, and labels might refer to the digit (0-9) or the color (8 options). CSL-Net achieved 98.24% and 99.02% accuracy on the two tasks. Remarkably, this is nearly identical to "Task Annotated" models (which had access to manual labels), which scored 99.48% and 99.61%.
  • Kaggle Fashion Product: A real-world dataset where images can be categorized by gender, category, or color. CSL-Net autonomously identified the three tasks and outperformed other semi-supervised baselines (like Pseudo-Labeling and SMiLE) by a wide margin, often achieving 60-70% higher accuracy.

Limitations and Future Work

While CSL represents a significant step forward, the authors note a few limitations:

  • Determining the Number of Tasks: Currently, the model requires repeated training runs with different task counts ($k$) to find the best fit. Future work will explore adding "low-quality constraints" to the Deconfusing-Net to automate the discovery of $k$.
  • Feature Learning: In complex image tasks, the model currently performs best when using a pre-trained CNN backbone. Learning basic visual features and high-level task concepts simultaneously from scratch remains a difficult open challenge.
  • Computational Cost: The iterative training process is inherently more computationally intensive than standard single-task training.

Summary updated

Abstract

Beyond machine learning’s success in the specific tasks, research for learning multiple tasks simultaneously is referred to as multi-task learning. However, existing multi-task learning needs manua...
BibTeX
@inproceedings{su2020task,
  author = {Xin Su and Yizhou Jiang and Shangqi Guo and Feng Chen},
  title = {{Task Understanding from Confusing Multi-task Data}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=fZMeCHhvzC},
}
Figure