Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels

Zixia Jia, Junpeng Li, Shichuan Zhang, Anji Liu, Zilong Zheng

Abstract

Traditional supervised learning heavily relies on human-annotated datasets, especially in data-hungry neural approaches. However, various tasks, especially multi-label tasks like document-level relation extraction, pose challenges in fully manual annotation due to the specific domain knowledge and large class sets. Therefore, we address the multi-label positive-unlabelled learning (MLPUL) problem, where only a subset of positive classes is annotated. We propose Mixture Learner for Partially Annotated Classification (MLPAC), an RL-based framework combining the exploration ability of reinforcement learning and the exploitation ability of supervised learning. Experimental results across various tasks, including document-level relation extraction, multi-label image classification, and binary PU learning, demonstrate the generalization and effectiveness of our framework.

Introduction: The Challenge of Partial Annotation

In an ideal machine learning scenario, every training instance is perfectly labeled with all applicable categories. However, real-world datasets are often "partially annotated." For instance, in Document-level Relation Extraction (DocRE), a human annotator might identify a "residence" relationship between two entities but overlook "place of birth" or "citizenship" due to the cognitive load of processing complex text and large label sets.

This leads to the Multi-Label Positive-Unlabelled (MLPUL) problem. Standard supervised learning models typically operate in a "negative mode," treating all unlabelled classes as negatives. This creates a significant bias: the model is penalized for correctly identifying positive relations that were simply missed by humans, leading to poor recall. While some methods attempt to mitigate this by estimating the "prior" (the expected ratio of positive labels), these estimates are notoriously difficult to obtain accurately. If the prior is misestimated, model performance degrades sharply.

Methodology: The MLPAC Framework

To address these challenges, we introduce MLPAC (Mixture Learner for Partially Annotated Classification). MLPAC treats multi-label classification as a one-step Markov Decision Process (MDP), effectively framing it as a contextual bandit problem. The framework utilizes an Actor-Critic architecture to balance exploration and exploitation.

The MLPAC framework architecture combining Policy and Critic networks with global and local rewards.

The framework consists of four primary components:

  1. Policy Network (Actor): Acts as the multi-label classifier, predicting "TRUE" or "FALSE" for each label.
  2. Critic Network: Evaluates the Policy's actions by providing a Local Reward based on the confidence of individual class predictions.
  3. Global Reward: To counteract the "negative mode" bias, a global reward based on recall is introduced. This encourages the Policy to explore and identify more potential positives rather than playing it safe with negative predictions.
  4. Iterative Training & Label Enhancement: The framework uses "label enhancement"—a form of pseudo-labeling where high-confidence predictions from both networks augment the training set for the Critic. This iterative process allows the Critic to provide increasingly accurate rewards.

By combining these elements, MLPAC leverages supervised learning for exploitation (learning from known labels) and reinforcement learning for exploration (discovering missing positive labels).

Experimental Setup

MLPAC was evaluated across three diverse domains to demonstrate its versatility:

  • NLP (Re-DocRED): A document-level relation extraction dataset with 97 relation types.
  • Computer Vision (MS-COCO): A multi-label image classification dataset with 80 object classes.
  • Binary PU Learning (CIFAR-10): Adapted into a binary "Positive vs. Unlabelled" task.

In these experiments, partial labeling was simulated by randomly hiding between 10% and 90% of the ground-truth positive annotations.

Key Insights

  • Prior-Agnosticism: Unlike many MLPUL methods that require a pre-defined estimate of missing labels, MLPAC is "prior agnostic." It dynamically learns the label distribution through the RL reward signal.
  • Discovery through Exploration: RL’s stochastic nature allows the agent to "experiment" with unlabelled classes. When the Critic confirms these experiments, the Policy is rewarded, effectively "discovering" labels missed by human annotators.
  • The Necessity of Global Reward: Ablation studies confirm that without the global recall-based reward, the model's recall drops significantly as it falls back into a conservative "negative bias."

Results

MLPAC demonstrated substantial performance gains, particularly in scenarios with extreme label sparsity.

Performance comparison (F1 score) across different annotation ratios on the Re-DocRED dataset.

On the Re-DocRED dataset with only 10% annotation, MLPAC achieved an F1 score of 66.0, vastly outperforming the standard "Negative Mode" baseline (7.3) and the previous state-of-the-art SSR-PU (48.6). Similarly, on MS-COCO with 30% labels, MLPAC reached an F1 of 77.0, significantly higher than other partial-label methods. Interestingly, the framework remained competitive even at 100% annotation, suggesting it can identify relations missed even in "complete" ground truth datasets.

Conclusion and Future Work

While MLPAC is highly effective, it currently treats label predictions as independent actions to avoid "correlation bias" from sparse data. However, real-world labels are often highly correlated (e.g., "sail boat" and "water"). Future research will focus on safely incorporating these dependencies into the RL framework and leveraging Large Language Models (LLMs) for more sophisticated label enhancement.

Summary updated

Abstract

Traditional supervised learning heavily relies on human-annotated datasets, especially in data-hungry neural approaches. However, various tasks, especially multi-label tasks like document-level relation extraction, pose challenges in fully manual annotation due to the specific domain knowledge and large class sets. Therefore, we address the multi-label positive-unlabelled learning (MLPUL) problem, where only a subset of positive classes is annotated. We propose Mixture Learner for Partially Annotated Classification (MLPAC), an RL-based framework combining the exploration ability of reinforcement learning and the exploitation ability of supervised learning. Experimental results across various tasks, including document-level relation extraction, multi-label image classification, and binary PU learning, demonstrate the generalization and effectiveness of our framework.
BibTeX
@inproceedings{jia2024combining,
  author = {Zixia Jia and Junpeng Li and Shichuan Zhang and Anji Liu and Zilong Zheng},
  title = {{Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=2GiCf6g1YI},
}
Figure