Topic-wise, Sentiment-wise, or Otherwise? Identifying the Hidden Dimension for Unsupervised Text Classification

Sajib Dasgupta, Vincent NgEMNLP 2009

Abstract

Unsupervised text clustering algorithms typically gravitate toward the most dominant feature in a dataset, which is usually the topic. This "greedy" behavior makes it difficult to cluster by "hidden" dimensions such as sentiment or author mood without significant manual labeling. We propose a novel framework that utilizes spectral clustering to expose multiple potential partitioning dimensions. By applying Maximum Margin Feature Ranking (MMFR), these dimensions are presented to the user as intuitive word summaries. A simple selection process then allows the user to steer the clustering toward the desired dimension. Experimental results across six datasets demonstrate that this minimal-feedback approach achieves high accuracy and stability, rivaling semi-supervised methods that require substantial labeled data.

Introduction

Traditional text clustering is inherently "greedy," prioritizing the strongest signal in the data. For instance, in a collection of movie reviews, a standard algorithm might group documents by genre (e.g., "sci-fi" vs. "romance") because plot-related vocabulary is more consistent than the varied expressions of sentiment. For a user seeking to sort reviews by positive or negative sentiment, this target dimension remains "hidden." Existing solutions to this problem generally fall into two categories: supervised learning, which requires large labeled datasets, or intensive human-in-the-loop feature engineering. This paper addresses the challenge of identifying these non-obvious dimensions with minimal human intervention.

Methods

The proposed method leverages the properties of Spectral Clustering, which uses the eigenvalues and eigenvectors of a data similarity matrix to uncover latent structures.

  1. Dimension Discovery: The algorithm computes the top eigenvectors of the data. In spectral clustering, each eigenvector represents a distinct way to partition the dataset.
  2. Feature Summarization: To avoid requiring users to read hundreds of documents, the authors employ Maximum Margin Feature Ranking (MMFR). A linear SVM is trained on the clusters produced by each eigenvector, and the words with the highest weights are extracted to create a summary of that dimension.
  3. Minimal User Feedback: The user is presented with word lists for the first few eigenvectors and simply selects the one that aligns with their goal (e.g., sentiment).
  4. Refinement: The algorithm identifies "unambiguous" documents—those far from the decision boundary—to finalize and refine the clustering.
Word summaries for the Books (BOO) and Electronics (ELE) datasets across eigenvectors e2 through e5. Sentiment-related clusters (e.g., e4 for BOO) are often found beyond the primary topic-based dimension (e2).

Key Insights

A critical finding is that sentiment is not always the primary dimension ($e_2$). In domains like Books or Electronics, the top eigenvector often captures the topic, while sentiment is "hidden" in the 3rd, 4th, or 5th eigenvectors. Furthermore, the study suggests that humans are significantly faster at identifying a dimension from a list of 10-20 informative words than from reading full documents. The algorithm also utilizes an "ambiguity" signal: documents with values near zero in an eigenvector are often sentimentally neutral or mixed. By temporarily excluding these during feature ranking, the system extracts a cleaner representation of the dimension.

Results

The approach was evaluated on six datasets: Politics (POL), Movie Reviews (MOV), and four Amazon domains (Books, DVDs, Electronics, Kitchen). Human evaluation showed high agreement (80% to 100%) on which eigenvector represented the sentiment dimension across all datasets.

Performance comparison of the proposed approach against baselines. The method shows significant stability and accuracy improvements, particularly in the Electronics (ELE) and Books (BOO) domains where standard baselines fail.

As shown in the results, while standard unsupervised baselines fluctuate—for example, the 2nd eigenvector baseline achieves only 50.8% accuracy on Electronics—the proposed method remains stable, reaching 65.8% to 70.9% accuracy. This performance is comparable to a Transductive SVM trained on an average of 275 manually labeled documents per dataset, highlighting the efficiency of the minimal-feedback model.

Conclusion and Future Work

While effective, the current method is limited to searching the top five eigenvectors. Subtle dimensions like sarcasm might reside in lower-order eigenvectors. Additionally, the current implementation focuses on binary partitions. Future work could extend this to multi-class hidden dimensions or incorporate interactive "nudging," where users can refine the dimension by adding or removing words from the summary lists.

Summary updated

BibTeX
@inproceedings{dasgupta2009topic,
  author = {Sajib Dasgupta and Vincent Ng},
  title = {{Topic-wise, Sentiment-wise, or Otherwise? Identifying the Hidden Dimension for Unsupervised Text Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
}
Figure