Dual-Label Learning With Irregularly Present Labels

Mingqian Li, Qiao Han, Yiteng Zhai, Ruifeng Li, Yao Yang, Hongyang Chen

Abstract

This paper introduces Dual-Label Learning (DLL), a framework designed to handle multi-task learning scenarios where labels are missing irregularly—a common challenge in scientific research. By treating two correlated labels as a bivariate system of equations and employing a novel "dual-tower" architecture, the authors demonstrate that partially labeled data can significantly improve model accuracy, even when up to 60% of the labels are missing.

The Problem: Irregularly Present Labels

In ideal multi-task datasets, every data point possesses a complete set of labels. However, in fields like drug discovery or materials science, labels are often "irregularly present." For instance, a drug molecule might be tested for one type of toxicity but not another due to budget or time constraints.

Traditional machine learning models typically handle this by either discarding samples with missing labels—which wastes valuable data—or using "shared-bottom" architectures that fail to explicitly model label correlations. The central challenge is maximizing the utility of these "semi-labeled" samples to improve predictions across all tasks.

Figure 5: (a) Illustration of irregularly present labels in drug discovery. (b) T-SNE visualization of feature distributions. (c) Comparison between Multi-task learning, Dual learning, and the proposed Dual-Label Learning (DLL).

The Dual-Label Learning (DLL) Framework

The authors propose shifting the perspective from viewing labels as independent targets to seeing them as a dual-function system. The DLL framework is built upon three core components:

  1. Dual-Tower Architecture: The model consists of two "towers." Tower 1 predicts label $y_1$ using features $x$ and label $y_2$ as an input. Tower 2 performs the inverse, predicting $y_2$ using $x$ and $y_1$.
  2. Three-Pronged Training:
    • Standard Supervision: Learning from fully labeled samples.
  3. Structural Duality: If only $y_1$ is known, the model predicts $y_2$, then uses that prediction to "reconstruct" $y_1$. This ensures the towers remain consistent with one another.
  4. Probabilistic Duality: A regularization term based on Bayes' Theorem aligns the joint probability distributions of the two towers.
  5. Alternate Inference: For samples where both labels are missing, the model performs an iterative process. It makes an initial guess for $y_1$, uses it to predict $y_2$, then uses that $y_2$ to refine $y_1$, repeating until the values converge.

Experimental Evaluation

The authors evaluated DLL on three distinct datasets:

  • Tox21: A real-world toxicity dataset (binary classification) with a 26% missing-label rate.
  • HIGGS: A synthetic high-energy physics dataset (regression) used to simulate varying missing-label rates.
  • MOF: A real-world materials science dataset (regression) focused on the synthesis conditions of metal-organic frameworks.

Key Results

  • Performance Gains: DLL outperformed baseline methods (such as Independent Decomposition and Label Stacking) by up to 9.6% in F1-score for classification and reduced error (MAPE) by up to 10.2% in regression tasks.
  • Extreme Robustness: DLL maintained high performance even when 60% of labels were missing. In several instances, DLL with 60% missing data outperformed baseline models that had access to 90% complete data.
  • Efficiency: Despite the iterative nature of alternate inference, the model typically reached equilibrium in fewer than 10 iterations, ensuring computational efficiency during deployment.

Insights and Future Work

The core innovation of DLL is treating labels as informative features for other labels. By formulating inference as solving a system of two equations ($y_1 = g(x, y_2)$ and $y_2 = f(x, y_1)$), the model can "triangulate" correct labels even from an unlabeled starting point.

While currently optimized for two labels, the authors suggest the framework can be extended to multi-label cases. Future research will focus on scaling to N-labels through generalized multi-tower architectures and providing formal mathematical proofs for convergence across arbitrary neural network configurations.

Summary updated

Abstract

In multi-task learning, we often encounter the case when the presence of labels across samples exhibits irregular patterns: samples can be fully labeled, partially labeled or unlabeled. Taking drug analysis as an example, multiple toxicity properties of a drug molecule may not be concurrently available due to experimental limitations. It triggers a demand for a new training and inference mechanism that could accommodate irregularly present labels and maximize the utility of any available label information. In this work, we focus on the two-label learning task, and propose a novel training and inference framework, Dual-Label Learning (DLL). The DLL framework formulates the problem into a dual-function system, in which the two functions should simultaneously satisfy standard supervision, structural duality and probabilistic duality. DLL features a dual-tower model architecture that explicitly captures the information exchange between labels, aimed at maximizing the utility of partially available labels in understanding label correlation. During training, label imputation for missing labels is conducted as part of the forward propagation process, while during inference, labels are regarded as unknowns of a bivariate system of equations and are solved jointly. Theoretical analysis guarantees the feasibility of DLL, and extensive experiments are conducted to verify that by explicitly modeling label correlation and maximizing the utility of available labels, our method makes consistently better predictions than baseline approaches by up to a 10% gain in F1-score or MAPE. Remarkably, our method provided with data at a label missing rate as high as 60% can achieve similar or even better results than baseline approaches at a label missing rate of only 10%.
BibTeX
@misc{li2024dual,
  author = {Mingqian Li and Qiao Han and Yiteng Zhai and Ruifeng Li and Yao Yang and Hongyang Chen},
  title = {{Dual-Label Learning With Irregularly Present Labels}},
  year = {2024},
  eprint = {2410.14380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.14380},
}
Figure