PanTop: Pandemic Topic Detection and Monitoring System (Student Abstract)

Yangxiao Bai, Kaiqun FuAAAI 2023 · DOI

Abstract

PanTop is a hybrid deep learning framework designed to track and visualize the evolution of COVID-19 discussions on Twitter. By fusing Latent Dirichlet Allocation (LDA) for hashtag modeling with BERT-based semantic embeddings for tweet content, the system maps how public sentiment and pandemic "storylines" shift across different temporal and geographic regions. Validation against real-world policy datasets demonstrates PanTop's effectiveness in bridging the gap between raw social media noise and actionable public health insights.

Introduction

During the COVID-19 pandemic, social media became a primary outlet for public discourse regarding vaccines, lockdowns, and mitigation policies. However, analyzing this "infodemic" is challenging due to the sheer volume of data and the informal nature of tweets, which often causes traditional models like LDA to underperform. Furthermore, public opinion is highly dynamic, shifting rapidly based on local policy announcements and geographic location. PanTop addresses these challenges by creating a system that detects topics and monitors their progression over time and space.

System Architecture and Methods

The PanTop system utilizes a multi-stage pipeline to process and cluster nearly half a million tweets. The framework is built on four primary pillars:

  1. Hybrid Feature Extraction: Recognizing that hashtags and tweet bodies serve different functions, the system applies LDA to capture structured hashtag information and BERT to generate semantic embeddings for tweet text.
  2. Autoencoder Fusion: To combine these distinct vector types without dimensionality explosion, an autoencoder "compresses" the weighted LDA and BERT vectors into a single synthesized representation.
  3. Spatiotemporal Clustering: The system employs Time Series KMeans for clustering. By treating time as a "z-axis" in a 2D projection, the model ensures that clusters respect the chronological flow of the pandemic.
  4. Storyline Generation: Topic evolution is visualized by calculating cosine similarity between topic pairs. High-similarity pairs are linked as "edges" in a directed graph, showing how one discussion topic transitions into another.
The PanTop system architecture, illustrating the flow from raw Twitter data through hybrid feature extraction and function services to the final visualization interface.

Datasets and Validation

The study utilizes two primary data sources: the IEEE DataPort COVID-19 Geo-tagged Tweets dataset (469,414 tweets from 2020–2022) and the CoronaNet Dataset of government policy responses. The authors used CoronaNet as a ground-truth benchmark, finding that detected topics (such as school regulations) correctly clustered in geographic regions like Kentucky and Nevada at the exact times those specific policies were enacted.

Key Results and Visualization

The system successfully identified distinct, high-relevance clusters. For example, "Topic 7" focused on social distancing and masks, while "Topic 1" centered on vaccines and quarantine measures. By calculating cosine similarity between individual tweets and cluster centroids, PanTop automatically selects the most representative tweet to serve as a human-readable label for each cluster.

Spatiotemporal storyline visualization showing the evolution of topics (e.g., from distancing to vaccines) and their geographic distribution across the United States.

One of the system's most powerful features is its ability to map regional relevance. The geographic distribution of tweet clusters allowed researchers to detect the localized impact of anti-epidemic isolation measures in states like Maryland and New Mexico directly from the data.

Conclusion and Future Work

While PanTop effectively handles spatiotemporal dynamics, the authors note that Twitter noise (slang, bots) and the brevity of tweets remain challenges for BERT-based context extraction. Future iterations of the system will integrate advanced text summarization techniques and attention mechanisms to better filter irrelevant noise and focus on the most informative components of public discourse.

Summary updated

BibTeX
@inproceedings{bai2023pantop,
  author = {Yangxiao Bai and Kaiqun Fu},
  title = {{PanTop: Pandemic Topic Detection and Monitoring System (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26939},
}
Figure