Abstract
Traditional text clustering methods often struggle to capture deep semantic relationships. This paper introduces the ADSL (Attention based on Dynamic Soft Labels) framework, which bridges the gap between unsupervised discovery and supervised feature extraction. By training a Densely Connected CNN (DCCNN) using iteratively refined "pseudo-labels" and a document-level attention mechanism, the model successfully handles noisy initial labels. Evaluation across three major datasets shows that ADSL consistently outperforms existing deep clustering baselines, achieving significant gains in accuracy and mutual information.
Introduction
Text clustering is a fundamental unsupervised task where documents are grouped without pre-existing labels. While traditional methods like TF-IDF are common, they often fail to grasp complex semantics. Supervised neural networks, such as CNNs, produce highly discriminative features but require ground-truth labels. The "dilemma" addressed here is how to leverage supervised-style architectures in an unsupervised setting. Simply using initial cluster assignments as "pseudo-labels" often leads to the "wrong labeling problem," where initial errors are amplified by the network.
The ADSL Framework
The proposed architecture, ADSL, consists of three integrated modules designed to refine feature extraction through a "pseudo-supervised" loop.
1. Soft Label Generation and Adjustment
Initially, documents are vectorized using an unsupervised model (e.g., an Autoencoder) and grouped via Agglomerative Hierarchical Clustering (AHC). Rather than assigning rigid "hard" labels, the system generates soft labels based on distances to cluster centroids. These labels are dynamic; every few epochs, they are updated by blending the original assignments with the model’s current predictions. This allows the labels to evolve as the model learns more discriminative features.
2. DCCNN with Document-Level Attention
The core of the feature extraction is a Densely Connected CNN (DCCNN), which captures local patterns across multiple scales. To combat the noise in pseudo-labels, the authors introduce a document-level attention mechanism.
This mechanism assigns an attention weight to each document. If a document's current features strongly conflict with its assigned pseudo-label, it is treated as "noisy" and assigned a lower weight in the loss function. This ensures that the model's learning is driven by the most reliable samples, effectively filtering out outliers that would otherwise distort the feature space.
Experimental Evaluation
The ADSL model was tested on three diverse datasets: Reuters10k (news), StackOverflow (technical titles), and the Fisher English corpus (transcribed speech).
Key Results
The NN+ADSL system consistently outperformed state-of-the-art models like DEC and STCC:
- Fisher Corpus: Achieved 91.31% Accuracy (ACC), compared to the 85.23% baseline.
- StackOverflow: Reached a 74.60% Normalized Mutual Information (NMI) score.
- Ablation Studies: Results confirmed that both dynamic updates (DSL) and the attention mechanism (ADSL) are critical for peak performance.
Discussion and Insights
The success of ADSL highlights several key principles in deep clustering:
- Softness and Dynamics: Soft labels prevent overfitting to early errors, while dynamic updates allow for a "coarse-to-fine" refinement of the data representation.
- Automated Quality Control: The attention mechanism acts as a filter, normalizing the influence of different clusters and preventing massive or noisy clusters from dominating the learning process.
Limitations
While effective, the model is sensitive to the hyperparameter $\alpha$ (initial label softness). Furthermore, the current implementation assumes a fixed number of clusters, suggesting that future work should focus on automated cluster number detection and reducing the computational overhead associated with iterative label updates.
Presented at Interspeech 2019.
Summary updated