Abstract
This paper presents a novel framework for tracking topic evolution in dynamic data streams by constructing a Semantic Connection Graph (SCG). By integrating Latent Dirichlet Allocation (LDA) with DeepWalk graph embeddings and density-based clustering, the approach overcomes the limitations of fixed topic counts and the consistency-sensitivity trade-off. Evaluated on Twitter data, the method demonstrates superior granularity in distinguishing similar events and adaptively identifying the number of topics without manual parameter tuning.
Introduction
Analyzing topic evolution in massive data streams like Twitter is challenging due to two primary issues: the requirement for a fixed number of topics ($k$) and the trade-off between temporal consistency and sensitivity to new events. Traditional models like LDA often produce disconnected topics across time slices or fail to react quickly to emerging trends. This paper proposes an adaptive and responsive system to address these flaws by treating topic evolution as a manifold learning problem.
Methods
The proposed framework, Semantic Connection Based Topic Evolution, follows a multi-stage pipeline:
- Sliding Window Semantics: LDA is applied within sliding windows to extract topic-document distributions ($\theta$).
- Semantic Connection Graph (SCG): A graph is constructed where documents are nodes. Edges are drawn based on the symmetrical KL divergence of their topic distributions, capturing both content similarity and temporal decay.
- Feature Extraction via DeepWalk: DeepWalk learns latent representations (Semantic Connection Features, or SCF) of the nodes by treating graph paths as sequences.
- Density-Based Clustering: A density-based algorithm identifies clusters in the SCF space. Unlike K-means, this method automatically determines the number of topics by identifying density peaks and filtering out noise.
Results and Discussion
The mapping of document similarities into a graph and subsequently into embeddings allows for a clear representation of complex semantic relationships in a 2D space. In this visualization, the physical distance between clusters directly represents semantic disparity, making it intuitive to observe how topics are related or distinct.
In head-to-head comparisons with On-Line LDA, the proposed method demonstrated superior granularity. While baseline models often lump multiple events into generic categories, the semantic connection method successfully separates them. For example, it identified specific, fleeting events such as Apple's hiring of a social media guru and various localized protests.
Conclusion
The Semantic Connection Based Topic Evolution method provides a robust, adaptive way to visualize and track shifting discourse in real-time data streams. While the qualitative results are strong, future work should focus on reducing computational overhead for true real-time streaming and developing standardized quantitative metrics for topic coherence and evolution tracking.
Summary updated