Introduction
Standard text co-clustering techniques often treat words as independent features, failing to account for the semantic relationships between different terms. This "semantic blind spot" can lead to poor performance, particularly when documents discussing the same topic use different vocabularies. To address this, we present WC-NMTF, a framework that regularizes Non-Negative Matrix Tri-Factorization (NMTF) with word co-occurrence data. By incorporating the distributional hypothesis—that words appearing in similar contexts share similar meanings—WC-NMTF ensures that semantically related words are grouped together, significantly enhancing both clustering accuracy and interpretability.
The Semantic Challenge in Text Mining
Traditional co-clustering methods like NMTF rely primarily on the document-word matrix. However, language diversity means that two documents might be topically identical yet share few common words (e.g., one using "galaxy" and another "nebula"). Standard models often fail to link these documents because they lack inherent knowledge of word relationships. WC-NMTF bridges this gap by leveraging word-word associations to guide the document-word factorization process.
Methodology
The WC-NMTF approach integrates semantic information through a joint optimization framework:
- Core NMTF Model: The document-word matrix $X$ is decomposed into three components: $Z$ (document clusters), $W$ (word clusters), and $S$ (a summary matrix linking the two).
- Semantic Regularization: A word co-occurrence matrix $M$ is constructed using Sparse Shifted Positive Point-wise Mutual Information (SPPMI). This matrix captures meaningful associations while filtering out noise.
- Joint Optimization: The word factor matrix $W$ is shared between the document-word factorization and the word-word factorization. This forces the model to find latent word representations that satisfy both the document structure and the semantic relationships in the co-occurrence data.
The authors derived efficient multiplicative update rules to solve this joint optimization, ensuring linear scalability with the number of non-zero entries and guaranteed convergence.
Experimental Setup
The model was evaluated on five benchmark datasets: CLASSIC4, CSTR, NG20, TREC, and LA Times. These datasets offer a range of challenges, including varying degrees of cluster overlap, sparsity, and balance, providing a robust testbed for the WC-NMTF framework.
Key Results and Insights
WC-NMTF consistently outperformed standard NMTF and other state-of-the-art variants across all tested datasets.
Improved Accuracy and Stability
On the diverse NG20 dataset, the Normalized Mutual Information (NMI) score improved from approximately 0.40 with standard NMTF to 0.63 with WC-NMTF. Furthermore, the model demonstrated remarkable stability across different values of the regularization parameter ($\lambda$), indicating that the semantic signal from word co-occurrences is a robust and reliable guide for clustering.
Enhanced Interpretability
A key advantage of the tri-factorization approach is the production of a "summary" matrix $S$, which explicitly maps word clusters to document clusters. To evaluate the quality of these word clusters, the authors measured their coherence against external knowledge (Wikipedia) using Point-wise Mutual Information (PMI) scores.
As shown in the results, WC-NMTF produced significantly more coherent word groups than standard NMTF. For instance, it successfully grouped related terms like "armenia/ottoman/turk" and "radio/amplifi/signal," which might remain disconnected in models that ignore co-occurrence data.
Conclusion and Future Work
WC-NMTF demonstrates that incorporating word-level semantic geometry is a powerful way to overcome data sparsity in text co-clustering. While the current model defines "context" at the document level, future work could explore more granular windows (e.g., sentences) or incorporate manifold regularization on the document side. The modular nature of this co-occurrence regularization suggests it could be broadly applied to other co-clustering architectures beyond NMTF.
Summary updated