Improved Training of Deep Text Clustering

Zonghao Yang, Wenpeng Hu, Yushan Tan, Zhunchen LuoEMNLP 2023 · OpenReview

Abstract

Deep clustering models often struggle with error propagation caused by learning from noisy, self-generated "pseudo-labels." This paper introduces DCGLU (Deep Clustering optimization from Generalized Labeled and Unlabeled data), a plug-and-play optimization method that treats low-confidence samples as unlabeled data. By applying Positive-Unlabeled (PU) learning principles, DCGLU breaks the cycle of error propagation, significantly boosting text clustering performance with negligible computational overhead.

Introduction

Deep text clustering aims to simultaneously learn feature representations and data groupings. In the absence of ground-truth labels, models rely on "generalized labels"—weak supervision signals derived from cluster assignments or sample similarities. However, these labels are inherently noisy, particularly in early training stages. Learning from erroneous labels leads to poor feature representations, which further degrades label quality. This feedback loop, known as error propagation, is a significant bottleneck for state-of-the-art systems.

The DCGLU Framework

The proposed DCGLU method addresses this by relaxing the requirement that every sample must have a definitive pseudo-label. Instead of forcing the model to trust every noisy assignment, DCGLU splits samples into high-confidence and low-confidence pools based on a confidence threshold $t$. High-confidence samples are treated as labeled (Positive), while low-confidence samples are treated as unlabeled.

Overview of the DCGLU framework, illustrating the flow from feature extraction to confidence-based splitting and PU-learning optimization.

The core of DCGLU is an adaptation of Positive-Unlabeled (PU) learning for multi-class clustering. By using a non-negative risk estimator, the model learns from the distribution of unlabeled data without being misled by specific noisy labels. DCGLU acts as a regularization term that can be integrated into existing deep clustering loss functions:

$$\mathcal{L}_{total} = \mathcal{L}_{original} + \lambda \cdot \hat{R}_{pu}(f)$$

Experimental Evaluation

The authors evaluated DCGLU on two widely used text benchmarks: SNIPS (a personal voice assistant dataset with 7 intent categories) and DBPedia (a subset of the DBPedia ontology with 14 classes). They compared DCGLU against a suite of baselines ranging from classical K-Means to modern deep systems like Deep Adaptive Clustering (DAC) and Deep Dirichlet Process Mixture (DeepDPM).

Key Results

Experimental results demonstrate that DCGLU consistently improves performance across all metrics, including Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Accuracy (ACC).

Performance comparison of various clustering methods on SNIPS and DBPedia datasets. Asterisks (*) denote models enhanced with DCGLU.

Key findings include:

  • State-of-the-Art Boosts: Adding DCGLU to the DeepDPM model pushed its accuracy on the SNIPS dataset from 88.35% to 92.07%. On DBPedia, DCGLU improved the DAC model's accuracy by nearly 4 percentage points (from 63.27% to 66.90%).
  • Noise Mitigation: Low-confidence samples are the primary source of feature space corruption. Treating them as unlabeled preserves their structural information without introducing label noise.
  • Universality and Efficiency: DCGLU is model-agnostic and improved every architecture tested. It scales linearly ($O(N)$) and typically adds less than 6% to total training time.

Conclusion and Future Work

DCGLU provides a robust, efficient solution to the error propagation problem in deep clustering by leveraging PU learning. While this study focused on text clustering, the authors plan to extend the framework to image and audio data, and explore automated methods for dynamic confidence threshold estimation.

Summary updated

Abstract

The classical deep clustering optimization methods basically leverage information such as clustering centers, mutual information, and distance metrics to construct implicit generalized labels to establish information feedback (weak supervision) and thus optimize the deep model. However, the resulting generalized labels have different degrees of errors in the whole clustering process due to the limitation of clustering accuracy, which greatly interferes with the clustering process. To this end, this paper proposes a general deep clustering optimization method from the perspective of empirical risk minimization, using the correlation relationship between the samples. Experiments on two classical deep clustering methods demonstrate the necessity and effectiveness of the method. Code is available at https://github.com/yangzonghao1024/DCGLU.
BibTeX
@inproceedings{yang2023improved,
  author = {Zonghao Yang and Wenpeng Hu and Yushan Tan and Zhunchen Luo},
  title = {{Improved Training of Deep Text Clustering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=sPB354cbmL},
}
Figure