Neural Text Clustering with Document-Level Attention Based on Dynamic Soft Labels

Zhi Chen, Wu Guo, Li-Rong Dai, Zhen-Hua Ling, Jun Du

Abstract

Traditional text clustering methods often struggle to capture deep semantic relationships. This paper introduces the ADSL (Attention based on Dynamic Soft Labels) framework, which bridges the gap between unsupervised discovery and supervised feature extraction. By training a Densely Connected CNN (DCCNN) using iteratively refined "pseudo-labels" and a document-level attention mechanism, the model successfully handles noisy initial labels. Evaluation across three major datasets shows that ADSL consistently outperforms existing deep clustering baselines, achieving significant gains in accuracy and mutual information.

Introduction

Text clustering is a fundamental unsupervised task where documents are grouped without pre-existing labels. While traditional methods like TF-IDF are common, they often fail to grasp complex semantics. Supervised neural networks, such as CNNs, produce highly discriminative features but require ground-truth labels. The "dilemma" addressed here is how to leverage supervised-style architectures in an unsupervised setting. Simply using initial cluster assignments as "pseudo-labels" often leads to the "wrong labeling problem," where initial errors are amplified by the network.

The ADSL Framework

The proposed architecture, ADSL, consists of three integrated modules designed to refine feature extraction through a "pseudo-supervised" loop.

The ADSL framework overview, illustrating the flow from word embeddings to soft label generation and model training.

1. Soft Label Generation and Adjustment

Initially, documents are vectorized using an unsupervised model (e.g., an Autoencoder) and grouped via Agglomerative Hierarchical Clustering (AHC). Rather than assigning rigid "hard" labels, the system generates soft labels based on distances to cluster centroids. These labels are dynamic; every few epochs, they are updated by blending the original assignments with the model’s current predictions. This allows the labels to evolve as the model learns more discriminative features.

2. DCCNN with Document-Level Attention

The core of the feature extraction is a Densely Connected CNN (DCCNN), which captures local patterns across multiple scales. To combat the noise in pseudo-labels, the authors introduce a document-level attention mechanism.

The document-level attention mechanism weighting the loss contribution of individual documents within clusters.

This mechanism assigns an attention weight to each document. If a document's current features strongly conflict with its assigned pseudo-label, it is treated as "noisy" and assigned a lower weight in the loss function. This ensures that the model's learning is driven by the most reliable samples, effectively filtering out outliers that would otherwise distort the feature space.

Experimental Evaluation

The ADSL model was tested on three diverse datasets: Reuters10k (news), StackOverflow (technical titles), and the Fisher English corpus (transcribed speech).

Key Results

The NN+ADSL system consistently outperformed state-of-the-art models like DEC and STCC:

  • Fisher Corpus: Achieved 91.31% Accuracy (ACC), compared to the 85.23% baseline.
  • StackOverflow: Reached a 74.60% Normalized Mutual Information (NMI) score.
  • Ablation Studies: Results confirmed that both dynamic updates (DSL) and the attention mechanism (ADSL) are critical for peak performance.

Discussion and Insights

The success of ADSL highlights several key principles in deep clustering:

  • Softness and Dynamics: Soft labels prevent overfitting to early errors, while dynamic updates allow for a "coarse-to-fine" refinement of the data representation.
  • Automated Quality Control: The attention mechanism acts as a filter, normalizing the influence of different clusters and preventing massive or noisy clusters from dominating the learning process.

Limitations

While effective, the model is sensitive to the hyperparameter $\alpha$ (initial label softness). Furthermore, the current implementation assumes a fixed number of clusters, suggesting that future work should focus on automated cluster number detection and reducing the computational overhead associated with iterative label updates.

Presented at Interspeech 2019.

Summary updated

Abstract

In this paper, the deep learning framework is applied in text clustering, an unsupervised task in natural language processing (NLP). Since there are no predefined labels available for text clustering, the deep neural network is trained in a pseudo-supervised fashion with labels generated from pre-clustering step. To address the wrong labelling problem from pre-clustering step, we adopt soft pseudo-labels instead of hard one-hot ones, and these labels are dynamically updated during training. Besides, we build a document-level attention over multiple documents based on dynamic soft pseudo-labels to further reduce the impact of the wrong labels. Experimental results on three public databases show that our model outperforms the state-of-the-art systems.
BibTeX
@inproceedings{chen2019neural,
  author = {Zhi Chen and Wu Guo and Li-Rong Dai and Zhen-Hua Ling and Jun Du},
  title = {{Neural Text Clustering with Document-Level Attention Based on Dynamic Soft Labels}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1417},
}
Figure