The Application of TF-IDF with Time Factor in the Cluster of Micro-blog Theme

Song Yu, Yang Wang, Tianchi Mo, Mingyang Liu, Hui Liu, Zhifang LiaoInternational Journal of Database Theory and Application 2017 · DOI

Abstract

Time factor is of great significance for the topic clustering for Micro-blog. Usually, the topics discussed most frequently during a certain period may become the hot issues. Therefore, this article has successfully obtained the method of TF-IDF-TF by different division of periods and setting of different weights, then applied it to the ULPIR Microblog content corpus, with the hierarchical clustering method and k-means method being used to make statistic analysis. The result of the experiment shows that, compared with the traditional TF-IDF (term frequency-inverse document frequency), the TF-IDF-TF method could provide more accurate clustering result, especially for specific topics during the period when users play most frequently.

Introduction

Micro-blogs (such as Weibo or Twitter) present a unique challenge for Natural Language Processing (NLP). Unlike static news articles, micro-blog posts are extremely short, highly informal, and deeply tied to real-time events. Standard TF-IDF (Term Frequency-Inverse Document Frequency) is a staple of keyword extraction, but it treats all documents in a corpus as equally important regardless of their timestamp. This paper argues that ignoring temporal context leads to "time-validity" loss, where niche but important topics discussed during peak hours are drowned out by persistent background noise.

Methodology

The authors propose a modified algorithm called TF-IDF-TF, where the second "TF" stands for Time Factor. The process involves several key stages:

  1. Temporal Segmentation: A 24-hour cycle is divided into six distinct periods based on user activity data. This includes three "active periods" and three "inactive periods."
  2. Weight Calculation: A weighting formula ($h_n$) assigns higher values to words appearing during active periods, proportional to the percentage of total daily activity in that window.
  3. The TF-IDF-TF Formula: Keywords are extracted for each period using standard TF-IDF, multiplied by the period's weight, and integrated into a global ranking.
  4. Mixed Clustering: To group topics, the researchers used a two-stage approach: Hierarchical Clustering to determine the optimal number of clusters ($K$) and initial centers, followed by K-means for efficient processing.
The TF-IDF-TF clustering process flowchart, illustrating the path from raw dataset through temporal weighting to final evaluation.

Key Insights

  • Activity as a Proxy for Relevance: A word's importance is tied to when it appears. A burst of activity during peak usage is a stronger signal of a "hot topic" than sporadic appearances during off-hours.
  • Preserving the "Long Tail": By applying time weights, the authors preserved specific keywords (like "iPhone" or "lottery") that are essential for identifying sub-topics but might otherwise be discarded as low-frequency noise.
  • Automating Cluster Selection: Using Hierarchical clustering as a pre-processor for K-means solves the common problem of choosing the number of clusters ($K$) beforehand, creating a less biased pipeline.

Results

The study utilized the NLPIR Micro-blog content corpus from February 2012, validating results against manual annotations and known hot topics like the "Wu Ying" legal case.

  • Higher Accuracy: The TF-IDF-TF method achieved a 31% increase in clustering accuracy over traditional TF-IDF.
  • Better Data Retention: The algorithm deleted significantly fewer records as outliers (260 vs. 382), indicating better recognition of meaningful patterns in noisy data.
  • Qualitative Superiority: TF-IDF-TF successfully captured event-related terms that traditional methods missed.
Visualization of clustering results for specific hot topics such as 'Wu Ying' and 'corruption' using the TF-IDF-TF method.

Limitations and Future Work

While the time factor provides a significant boost, the model currently treats feature words as independent entities and does not account for polysemy (words with multiple meanings). Future research will likely involve integrating semantic analysis to better handle the linguistic nuances and slang prevalent in social media.

Summary updated

Abstract

Time factor is of great significance for the topic clustering for Micro-blog. Usually, the topics discussed most frequently during a certain period may become the hot issues. Therefore, this article has successfully obtained the method of TF-IDF-TF by different division of periods and setting of different weights, then applied it to the ULPIR Microblog content corpus, with the hierarchical clustering method and k-means method being used to make statistic analysis. The result of the experiment shows that, compared with the traditional TF-IDF( term frequency-inverse document frequency ), the TF-IDF-TF method could provide more accurate clustering result, especially for specific topics during the period when users play most frequently.
BibTeX
@article{yu2017application,
  author = {Song Yu and Yang Wang and Tianchi Mo and Mingyang Liu and Hui Liu and Zhifang Liao},
  title = {{The Application of TF-IDF with Time Factor in the Cluster of Micro-blog Theme}},
  journal = {International Journal of Database Theory and Application},
  year = {2017},
  doi = {10.14257/ijdta.2017.10.2.03},
}
Figure