Calibrating Pseudo-Labeling with Class Distribution for Semi-supervised Text Classification

Weiyi Yang, Richong Zhang, Junfan Chen, Jiawei Sheng2025 · DOI

Abstract

Semi-supervised text classification (SSTC) is a powerful technique for leveraging large volumes of unlabeled data when labeled examples are scarce. However, standard pseudo-labeling approaches often fail when unlabeled data is imbalanced, leading to a feedback loop where models over-predict majority classes. We introduce PL-POT (Prototypical OT-guided Pseudo-Labeling), a framework that reformulates pseudo-labeling as an Optimal Transport (OT) problem. By dynamically estimating class distributions via a Memory Bank and aligning samples with class prototypes, PL-POT effectively mitigates bias and achieves state-of-the-art performance across multiple benchmarks, even in scenarios with extreme class imbalance.

Introduction

In semi-supervised text classification, the model typically generates "pseudo-labels" for unlabeled data to expand its training set. A significant challenge arises when the unlabeled data distribution is highly imbalanced—a common occurrence in real-world settings. If a model develops a slight initial bias toward a majority class, it will assign more pseudo-labels to that class, reinforcing the bias in a process known as "catastrophic error propagation."

Comparison of pseudo-labeling approaches: (a) Initial step, (b) Existing methods showing majority class bias, and (c) PL-POT using prototypes to align with the true distribution.

Existing solutions often assume that the labeled and unlabeled data share the same distribution, which is rarely true. As shown in Figure 1, standard methods tend to collapse toward majority classes, whereas the proposed method uses class prototypes and distribution constraints to maintain a balanced and accurate representation of the data.

Methodology: The PL-POT Framework

PL-POT addresses the limitations of confidence-based pseudo-labeling by treating the assignment process as a global alignment problem rather than a series of independent local decisions. The framework consists of four primary modules:

The PL-POT architecture: (a) Backbone classifier, (b) Class distribution estimation, (c) Prototypical coupling cost measurement, and (d) OT-guided pseudo-labeling.

1. Optimal Transport (OT) for Labeling

Instead of simply picking the highest-probability class for each sample, PL-POT "transports" the distribution of a batch of unlabeled samples to a target class distribution. This formulation forces the model to respect global class frequencies, preventing it from over-assigning labels to a single dominant category.

2. Dynamic Class Distribution Estimation

Because the true distribution of unlabeled data is unknown, the model employs a Memory Bank (MB). This bank stores recent high-confidence pseudo-labels and ground-truth labels to dynamically estimate the global class distribution on the fly. This estimate serves as the target for the OT process, allowing the model to adapt to the specific imbalance of the dataset.

3. Prototypical Coupling Cost

To determine the best label for a sample during OT, the model calculates the semantic similarity between the sample's embedding and "class prototypes"—the mean embeddings of classes stored in the Memory Bank. This ensures that labels are assigned based on semantic relevance rather than just classifier confidence.

4. Mistrust Weighting

The authors introduce a "mistrust degree" ($\alpha$). If the classifier’s raw prediction significantly disagrees with the label assigned by the OT process, the model reduces that sample's weight in the loss function. This mechanism filters out noisy pseudo-labels that could otherwise corrupt the training process.

Experimental Evaluation

Datasets and Setup

The framework was evaluated on four major benchmarks: AGNews (4 classes), Yahoo (10 classes), DBpedia (14 classes), and Banking (77 classes). To test robustness, the researchers simulated various imbalance ratios ($\tau_u$), where the most frequent class appeared up to 20 times more often than the least frequent one.

Key Results

  • State-of-the-Art Performance: On the AGNews dataset with a 20:1 imbalance ratio, PL-POT achieved an F1-score of 84.08%, outperforming previous leading methods like MixText (82.75%) and PGPL (81.58%).
  • Ablation Studies: Removing the Distribution Estimation (DE) module led to a 23% drop in F1-score on AGNews, highlighting that accurate distribution tracking is the critical "secret sauce" of the framework.
  • Efficiency vs. Scale: PL-POT demonstrated that task-specific calibration can allow smaller models to outperform massive Large Language Models (LLMs) like LLaMA2 and GPT-4 on these classification tasks, offering a more cost-effective and accurate solution.

Conclusion

PL-POT shifts the paradigm of pseudo-labeling from simple confidence thresholds to distribution-aware alignment. By using Optimal Transport and dynamic estimation, it successfully navigates the pitfalls of imbalanced semi-supervised learning. Future work will explore extending this OT-based calibration to other domains, such as image classification, and investigating its performance in open-set scenarios where unlabeled data may contain unknown classes.

Summary updated

BibTeX
@misc{yang2025calibrating,
  author = {Weiyi Yang and Richong Zhang and Junfan Chen and Jiawei Sheng},
  title = {{Calibrating Pseudo-Labeling with Class Distribution for Semi-supervised Text Classification}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.658},
}
Figure