Abstract
Time factor is of great significance for the topic clustering for Micro-blog. Usually, the topics discussed most frequently during a certain period may become the hot issues. Therefore, this article has successfully obtained the method of TF-IDF-TF by different division of periods and setting of different weights, then applied it to the ULPIR Microblog content corpus, with the hierarchical clustering method and k-means method being used to make statistic analysis. The result of the experiment shows that, compared with the traditional TF-IDF (term frequency-inverse document frequency), the TF-IDF-TF method could provide more accurate clustering result, especially for specific topics during the period when users play most frequently.
Introduction
Micro-blogs (such as Weibo or Twitter) present a unique challenge for Natural Language Processing (NLP). Unlike static news articles, micro-blog posts are extremely short, highly informal, and deeply tied to real-time events. Standard TF-IDF (Term Frequency-Inverse Document Frequency) is a staple of keyword extraction, but it treats all documents in a corpus as equally important regardless of their timestamp. This paper argues that ignoring temporal context leads to "time-validity" loss, where niche but important topics discussed during peak hours are drowned out by persistent background noise.
Methodology
The authors propose a modified algorithm called TF-IDF-TF, where the second "TF" stands for Time Factor. The process involves several key stages:
- Temporal Segmentation: A 24-hour cycle is divided into six distinct periods based on user activity data. This includes three "active periods" and three "inactive periods."
- Weight Calculation: A weighting formula ($h_n$) assigns higher values to words appearing during active periods, proportional to the percentage of total daily activity in that window.
- The TF-IDF-TF Formula: Keywords are extracted for each period using standard TF-IDF, multiplied by the period's weight, and integrated into a global ranking.
- Mixed Clustering: To group topics, the researchers used a two-stage approach: Hierarchical Clustering to determine the optimal number of clusters ($K$) and initial centers, followed by K-means for efficient processing.
Key Insights
- Activity as a Proxy for Relevance: A word's importance is tied to when it appears. A burst of activity during peak usage is a stronger signal of a "hot topic" than sporadic appearances during off-hours.
- Preserving the "Long Tail": By applying time weights, the authors preserved specific keywords (like "iPhone" or "lottery") that are essential for identifying sub-topics but might otherwise be discarded as low-frequency noise.
- Automating Cluster Selection: Using Hierarchical clustering as a pre-processor for K-means solves the common problem of choosing the number of clusters ($K$) beforehand, creating a less biased pipeline.
Results
The study utilized the NLPIR Micro-blog content corpus from February 2012, validating results against manual annotations and known hot topics like the "Wu Ying" legal case.
- Higher Accuracy: The TF-IDF-TF method achieved a 31% increase in clustering accuracy over traditional TF-IDF.
- Better Data Retention: The algorithm deleted significantly fewer records as outliers (260 vs. 382), indicating better recognition of meaningful patterns in noisy data.
- Qualitative Superiority: TF-IDF-TF successfully captured event-related terms that traditional methods missed.
Limitations and Future Work
While the time factor provides a significant boost, the model currently treats feature words as independent entities and does not account for polysemy (words with multiple meanings). Future research will likely involve integrating semantic analysis to better handle the linguistic nuances and slang prevalent in social media.
Summary updated