Learning Distributed Representations and Deep Embedded Clustering of Texts

Shuang Wang, Amin Beheshti, Yufei Wang, Jianchao Lu, Quan Z. Sheng, Stephen Elbourn, Hamid Alinejad-Rokny

TL;DR

This paper tackles the challenge of reducing instructor workload in grading large volumes of text-based student assessments. They propose a novel deep embedded clustering framework that leverages pre-trained language models (like BERT) for text representation and contrastive learning to improve the quality of these representations, ultimately leading to more effective clustering of similar assessments.

What Problem Does This Paper Address?

Grading student assessments, especially in large online courses, is a time-consuming and labor-intensive task for instructors. Manually reviewing and grading hundreds or thousands of essays, reports, or code submissions is a significant bottleneck. The paper identifies a crucial need for automated methods to streamline this process. While clustering similar assessments has been proposed as a solution, existing methods often struggle with the high dimensionality of text data and may not effectively capture nuanced similarities. This paper aims to bridge this gap by developing a robust deep learning-based clustering approach specifically for text assessments.

Methods

The core of the proposed solution is a deep embedded clustering framework that integrates several key components:

  • Distributed Text Representation: The paper utilizes pre-trained language models (PLMs), specifically mentioning BERT and DistilBERT, to transform text assessments into dense vector representations. This is a standard approach for capturing semantic meaning in text.
  • Contrastive Learning: To enhance the quality of these vector representations, the authors incorporate contrastive learning. This technique aims to pull representations of similar texts (e.g., augmented versions of the same assessment) closer together in the embedding space while pushing dissimilar texts further apart. This helps the model learn representations that better distinguish subtle differences between similar documents.
  • Deep Embedded Clustering: The framework combines the learned representations with clustering algorithms. They explore three strategies:
  • Label as Representation: This approach projects the text embeddings into a space where the dimensionality equals the number of clusters, and the highest value indicates the cluster assignment.
  • K-means: A classic and widely used clustering algorithm.
  • Self-Organizing Map (SOM): A neural network-based clustering algorithm known for its ability to create spatially organized representations.
  • General Framework: The paper proposes a general framework that can accommodate different clustering algorithms within the deep embedded clustering pipeline.

Datasets

The proposed framework was evaluated using two real-world datasets:

  • AgNews Dataset: A collection of news titles categorized into four topics. This dataset is commonly used in data mining tasks like clustering and classification.
  • StackOverflow Dataset: A subset of challenge data containing question titles from StackOverflow, categorized into 20 different topics. This dataset is used for analyzing text clustering performance.

Key Insights and Ideas

  • Contrastive Learning for Nuanced Similarity: The paper highlights that simply representing texts as vectors might not be enough. Contrastive learning is crucial for learning representations that can effectively distinguish between subtly different but semantically related assessments, which is vital for accurate clustering.
  • PLMs as Powerful Text Encoders: The effectiveness of pre-trained language models like BERT in generating high-quality text embeddings is reaffirmed, forming a strong foundation for the clustering task.
  • SOM and K-means Outperform Label Representation: The experimental results suggest that traditional clustering algorithms like K-means and SOM, when combined with deep representations, generally perform better than the "label as representation" strategy for this task. This indicates that leveraging established clustering algorithms can be more effective than a direct projection approach.
  • Importance of Hyperparameter Tuning: The experiments demonstrate that parameters like learning rate and temperature in contrastive learning significantly impact clustering performance, emphasizing the need for careful calibration.

Key Results

The paper evaluates the performance of their proposed framework using Accuracy (ACC) and Normalized Mutual Information (NMI) metrics.

  • AgNews Dataset:
    • The SOM algorithm achieved the highest NMI (0.824) and ACC (0.947) at a learning rate of 10⁻⁵.
  • K-means also performed strongly, with an NMI of 0.822 and ACC of 0.944 at the same learning rate.
  • When comparing learning rate scales, K-means achieved the highest NMI (0.904) and ACC (0.974) at a scale of 500.
  • For temperature calibration, SOM achieved the highest NMI (0.966) and ACC (0.992) at a temperature of 0.9.
  • StackOverflow Dataset:
    • K-means generally performed best or comparably to SOM across various experiments. For instance, K-means achieved the highest NMI (0.667) and ACC (0.711) at a learning rate of 10⁻⁶.
  • The experiments showed that SOM and K-means algorithms consistently outperformed their "R" variants (SOMR and KmeansR), which likely represent variations in how representations are handled.

Overall, the results indicate that the proposed deep embedded clustering framework, particularly with SOM and K-means, effectively clusters text assessments, demonstrating significant improvements over simpler representation-based methods.

Limitations and Future Work

  • Computational Resources: The paper acknowledges that BERT, while effective, consumes significant server resources. They mention using DistilBERT as a more efficient alternative.
  • Hyperparameter Sensitivity: The performance is shown to be sensitive to hyperparameters like learning rate and temperature, requiring careful tuning.
  • Future Work: The authors plan to further investigate the relationships among different texts within a cluster and explore constructing and studying different relationship graphs for texts. This suggests a direction towards more sophisticated analysis of the clustered content.

Summary updated

Abstract

Instructors face significant time and effort constraints when grading students’ assessments on a large scale. Clustering similar assessments is a unique and effective technique that has the potential to significantly reduce the workload of instructors in online and large-scale learning environments. By grouping together similar assessments, marking one assessment in a cluster can be scaled to other similar assessments, allowing for a more efficient and streamlined grading process. To address this issue, this paper focuses on text assessments and proposes a method for reducing the workload of instructors by clustering similar assessments. The proposed method involves the use of distributed representation to transform texts into vectors, and contrastive learning to improve the representation that distinguishes the differences among similar texts. The paper presents a general framework for clustering similar texts that includes label representation, K-means, and self-organization map algorithms, with the objective of improving clustering performance using Accuracy (ACC) and Normalized Mutual Information (NMI) metrics. The proposed framework is evaluated experimentally using two real datasets. The results show that self-organization maps and K-means algorithms with Pre-trained language models outperform label representation algorithms for different datasets.
BibTeX
@inproceedings{wang2023learning,
  author = {Shuang Wang and Amin Beheshti and Yufei Wang and Jianchao Lu and Quan Z. Sheng and Stephen Elbourn and Hamid Alinejad-Rokny},
  title = {{Learning Distributed Representations and Deep Embedded Clustering of Texts}},
  booktitle = {Algorithms},
  year = {2023},
  url = {https://openreview.net/forum?id=872F2P0zZ5},
}
Figure