Dual-Label Learning With Irregularly Present Labels

Mingqian Li, Qiao Han, Ruifeng Li, Yao Yang, Hongyang Chen

TL;DR

This paper introduces Dual-Label Learning (DLL), a novel framework designed for multi-task learning when labels are missing irregularly—a common headache in scientific research. By explicitly modeling bi-directional correlations between labels through a "dual-tower" architecture and an iterative inference process, DLL achieves up to a 10% performance boost over standard methods and remains remarkably robust even when 60% of the data is unlabeled.

What Problem Does This Paper Address?

In fields like drug discovery or high-energy physics, datasets are rarely "clean." You might have a molecule tested for one toxic property ($y_1$) but not another ($y_2$) due to budget or time constraints. This leads to irregularly present labels: some samples are fully labeled, some are partially labeled (semi-labeled), and some are completely unlabeled.

Traditional multi-task learning (MTL) usually assumes every sample has a full set of labels during training. When labels are missing, these models often struggle to capture the underlying relationships between tasks. While some methods use matrix completion or graph neural networks to fill the gaps, they are often limited to specific classification tasks and don't generalize well to regression or scenarios with very sparse data. The authors set out to create a training and inference mechanism that treats missing labels not just as a nuisance, but as an opportunity to leverage cross-label correlations.

Comparison of learning mechanisms: (c1) standard multi-task learning vs. (c3) the proposed Dual-Label Learning.

Methods

The core of DLL is a dual-function system. Instead of predicting $y_1$ and $y_2$ independently from features $x$, DLL learns two interdependent functions:

  1. A primal function $f(x, y_1) \to y_2$
  2. A dual function $g(x, y_2) \to y_1$

This setup is implemented via a dual-tower model architecture. The two towers share a "bottom" for feature extraction from $x$ but have separate branches that take the other label as an additional input.

The dual-tower architecture showing how information flows between labels during training and inference.

Training

DLL optimizes a multi-objective loss function that balances three types of duality:

  • Standard Supervision: Learning from fully labeled samples.
  • Structural Duality: Using semi-labeled samples to minimize "reconstruction loss." If you only have $y_1$, the model predicts $y_2$, then uses that prediction to try and reconstruct the original $y_1$.
  • Probabilistic Duality: A regularization term based on Bayes' Theorem that ensures the joint distribution $P(y_1, y_2 | x)$ is consistent regardless of which label you start with.

Inference

  • Semi-labeled samples: If one label is known, the model uses "direct inference" to predict the missing one.
  • Unlabeled samples: The model treats $y_1$ and $y_2$ as unknowns in a bivariate system of equations and solves them using alternate inference—an iterative process where predictions for $y_1$ and $y_2$ are refined until they reach an equilibrium.

Datasets

The authors evaluated DLL across three diverse domains:

  1. Tox21: A real-world drug toxicity dataset (binary classification) with a natural 26% missing label rate.
  2. HIGGS: A synthetic high-energy physics dataset (regression) where missingness was manually introduced.
  3. MOF: A materials science dataset for metal-organic framework synthesis (regression).

Key Insights and Ideas

  • Explicit is Better than Implicit: Standard MTL models label correlations indirectly through shared layers. DLL models them explicitly by making one label a feature for the other. This "feedback loop" is the secret sauce for handling missing data.
  • Robustness to Sparsity: One of the most striking findings is that DLL at a 60% missing rate can outperform baseline models that have only 10% missing data. This suggests that the model is exceptionally good at "hallucinating" missing information by understanding the deep ties between labels.
  • Convergence is Fast: Despite being an iterative process, the alternate inference scheme typically converges in fewer than 10 iterations, making it computationally efficient for real-time deployment.

Key Results

  • Accuracy Gains: DLL showed a consistent edge, achieving up to a 9.6% gain in F1-score on Tox21 and a 10.2% reduction in Mean Absolute Percentage Error (MAPE) on the HIGGS dataset compared to the best baselines.
  • Trend Analysis: As the missing rate increases, the performance gap between DLL and traditional methods (like Independent Decomposition or simple Multi-Task Learning) widens significantly, particularly for "double-label" prediction tasks where both labels are missing.
MAPE trends on the HIGGS dataset showing DLL's superior performance across various missing rates.

Limitations and Future Work

While powerful, the current DLL framework is primarily optimized for two-label scenarios. While the authors suggest it can be adapted for more labels by enumerating pairs, this could become computationally expensive and might fail to capture higher-order correlations (e.g., how $y_1, y_2,$ and $y_3$ interact simultaneously).

Future research directions include:

  • Developing more sophisticated algorithms to handle an arbitrary number of labels without pairwise enumeration.
  • Establishing formal theoretical guarantees for the convergence of the alternate inference scheme as the number of unknown labels grows.
  • Expanding the framework to handle more complex data modalities like graphs or time-series data more natively.

Summary updated

Abstract

In multi-task learning, labels are often missing irregularly across samples, which can be fully labeled, partially labeled or unlabeled. The irregular label presence often appears in scientific studies due to experimental limitations. It triggers a demand for a new training and inference mechanism that could accommodate irregularly present labels and maximize their utility. This work focuses on the two-label learning task and proposes a novel training and inference framework, Dual-Label Learning (DLL). The DLL framework formulates the problem into a dual-function system, in which the two functions should simultaneously satisfy standard supervision, structural duality and probabilistic duality. DLL features a dual-tower model architecture that allows for explicit information exchange between labels, aimed at maximizing the utility of partially available labels. During training, missing labels are imputed as part of the forward propagation process, while during inference, labels are predicted jointly as unknowns of a bivariate system of equations. Our theoretical analysis guarantees the feasibility of DLL, and extensive experiments are conducted to verify that by explicitly modeling label correlation and maximizing label utility, our method makes consistently better prediction than baseline approaches by up to 9.6% gain in F1-score or 10.2% reduction in MAPE. Remarkably, DLL maintains robust performance at a label missing rate of up to 60%, achieving even better results than baseline approaches at lower missing rates down to only 10%.
BibTeX
@misc{li2024dual,
  author = {Mingqian Li and Qiao Han and Ruifeng Li and Yao Yang and Hongyang Chen},
  title = {{Dual-Label Learning With Irregularly Present Labels}},
  year = {2024},
  eprint = {2410.14380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.14380},
}
Figure