Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels

Zixia Jia, Junpeng Li, Shichuan Zhang, Anji Liu, Zilong Zheng

Abstract

Traditional supervised learning heavily relies on human-annotated datasets, especially in data-hungry neural approaches. However, various tasks, especially multi-label tasks like document-level relation extraction, pose challenges in fully manual annotation due to the specific domain knowledge and large class sets. Therefore, we address the multi-label positive-unlabelled learning (MLPUL) problem, where only a subset of positive classes is annotated. We propose Mixture Learner for Partially Annotated Classification (MLPAC), an RL-based framework combining the exploration ability of reinforcement learning and the exploitation ability of supervised learning. Experimental results across various tasks, including document-level relation extraction, multi-label image classification, and binary PU learning, demonstrate the generalization and effectiveness of our framework.

The Challenge of Partial Labels

In real-world scenarios, perfectly labeled datasets are rare. For complex tasks such as Document-level Relation Extraction (DocRE) or large-scale image tagging, human annotators often fail to identify all positive labels due to the high volume of classes or contextual complexity. This leads to the Multi-Label Positive-Unlabelled (MLPUL) problem: the training set contains some confirmed positive labels, but the remaining "unlabeled" data is a mixture of true negatives and missing positives.

Standard Supervised Learning (SL) typically treats all unlabeled data as negative (the "Negative Mode"). This conservative approach results in models with high precision but very low recall, as they struggle to predict any relationship or tag not explicitly seen during training. While existing Positive-Unlabelled (PU) methods attempt to mitigate this, they often require a "prior"—an accurate estimate of the positive-to-negative label ratio—which is frequently unavailable in practice.

MLPAC: A Mixture Learner Approach

To overcome these limitations, we introduce MLPAC (Mixture Learner for Partially Annotated Classification). MLPAC treats multi-label classification as a one-step Markov Decision Process (MDP) and utilizes an actor-critic reinforcement learning (RL) framework to balance exploration and exploitation.

The framework consists of two primary components:

  1. The Policy Network (Actor): Acts as the multi-label classifier, predicting the probability of each class.
  2. The Critic Network: Evaluates the policy's predictions and provides a Local Reward for each class, guiding the policy to discover latent patterns in the unlabeled space.
The MLPAC framework architecture, illustrating the interaction between the Policy Network (Actor) and Critic Network through global and local rewards.

The model is optimized using a dual-reward system:

  • Global Reward: Calculated based on a recall function against known partial labels. This ensures the model captures the ground-truth positives and prevents it from defaulting to a "negative-only" prediction state.
  • Local Reward: Generated by the critic network, allowing the model to explore the "unknown" space and identify potential positive labels that were missed during initial annotation.

Through an iterative Label Enhancement process, the policy and critic collaborate to identify high-confidence "pseudo-labels," which are then used to refine the critic in a continuous cycle of discovery.

Experimental Evaluation and Key Results

We evaluated MLPAC across three distinct domains: Document-level Relation Extraction (Re-DocRED), multi-label image classification (MS-COCO), and binary PU learning (CIFAR-10).

Performance Gains under Label Scarcity

MLPAC demonstrated significant improvements, particularly when annotations were extremely sparse. On the Re-DocRED task with only 10% of positive labels available, MLPAC achieved an F1 score of 66.0, a dramatic increase from the 7.3 F1 score achieved by the standard "Negative Mode" baseline.

Performance comparison (F1 score) of MLPAC against various baselines across different annotation ratios.

Prior-Agnostic Robustness

A key advantage of MLPAC is its "prior-agnostic" nature. Unlike state-of-the-art PU methods like SSR-PU, which rely on accurate label ratio estimations, MLPAC dynamically learns the label distribution. This makes it far more robust for real-world applications where the true ground truth is unknown.

Cross-Domain Versatility

The framework's success was not limited to NLP. It performed competitively in image classification tasks on MS-COCO and binary PU tasks on CIFAR-10, proving its effectiveness as a general-purpose solution for partial label problems.

Conclusion and Future Directions

MLPAC successfully bridges the gap between supervised learning and reinforcement learning to solve the MLPUL problem. By leveraging RL's exploration capabilities, the model can discover missing positive labels without requiring prior knowledge of label distributions.

While the current model treats label predictions as independent actions, future work could explore integrating label correlations to further enhance precision. Additionally, addressing the computational complexity inherent in iterative actor-critic training remains an area for optimization.

Summary updated

Abstract

Traditional supervised learning heavily relies on human-annotated datasets, especially in data-hungry neural approaches. However, various tasks, especially multi-label tasks like document-level relation extraction, pose challenges in fully manual annotation due to the specific domain knowledge and large class sets. Therefore, we address the multi-label positive-unlabelled learning (MLPUL) problem, where only a subset of positive classes is annotated. We propose Mixture Learner for Partially Annotated Classification (MLPAC), an RL-based framework combining the exploration ability of reinforcement learning and the exploitation ability of supervised learning. Experimental results across various tasks, including document-level relation extraction, multi-label image classification, and binary PU learning, demonstrate the generalization and effectiveness of our framework.
BibTeX
@inproceedings{jia2024combining,
  author = {Zixia Jia and Junpeng Li and Shichuan Zhang and Anji Liu and Zilong Zheng},
  title = {{Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=2GiCf6g1YI},
}
Figure