Abstract
This paper explores the application of Latent Dirichlet Allocation (LDA) and Gibbs sampling to extract and visualize key themes from a corpus of political emails. By leveraging parallel computing via the BashReduce tool, the study achieved a 30% speedup in data processing. Experimental results demonstrate that the LDA-based approach significantly outperforms traditional TF-IDF models in precision when identifying target topic words, providing a more nuanced thematic understanding than simple frequency-based methods.
Introduction
As digital communication continues to expand, Social Network Analysis (SNA) has become essential for understanding information flow within specific groups. Traditional "unigram" approaches, which rely on simple word counts, often fail to capture the nuance of documents containing multiple overlapping themes. This research addresses the challenge of characterizing sender preferences—specifically emails from Obama.com to residents of Columbus, Ohio—by moving beyond keyword frequency to a sophisticated topic modeling approach that automatically categorizes emails into latent themes like healthcare, local news, and campaign contributions.
Methodology
The core of the research rests on three pillars:
Latent Dirichlet Allocation (LDA)
LDA is a hierarchical Bayesian model that assumes each document is a mixture of various topics, and each topic is a distribution of words. This allows for a much more flexible and accurate representation of text than simpler models.
Gibbs Sampling
While many LDA implementations use Expectation-Maximization (EM), this study utilizes Gibbs sampling, a Markov Chain Monte Carlo algorithm. It iteratively updates the topic assignment for each word to converge on the most likely distribution. Experimental trials determined that the process reaches a stable state after approximately 500 iterations, regardless of initial values.
Parallel Computing with BashReduce
To handle the computational load of word counting, the authors implemented BashReduce, a tool that brings MapReduce functionality to a standard bash environment. This allowed processing to be split across multiple instances, significantly reducing execution time.
Experimental Setup
The study utilized a specialized dataset of 58 emails sent from Obama.com to accounts registered in Columbus, Ohio. These emails were collected via Datagreening.com, a green server infrastructure developed to study the energy footprint of cloud computing. The final processed corpus consisted of a vocabulary of 1,118 unique words.
Results and Discussion
Efficiency Gains
Using BashReduce on two nodes resulted in a 30% speedup, reducing processing time from 5 minutes 18 seconds (local execution) to 3 minutes 41 seconds. This demonstrates the scalability of the approach even on a small cluster.
Qualitative Topic Analysis
The LDA model successfully identified coherent thematic clusters. For example, Topic 1 clearly clustered words related to healthcare, while Topic 3 focused on local elements like "Ohio" and "voter."
Quantitative Performance
The LDA model was compared against a TF-IDF baseline. When the size of the topic word list ($|TW|$) was set to 15, the LDA model achieved a precision rate 53.96% higher than the TF-IDF model. Furthermore, the results showed that as the size of the target word list increased, the predictive power of the LDA model grew significantly stronger than the baseline.
Conclusion and Future Work
The primary limitation identified is the computational cost of updates, as adding new documents currently requires re-running the entire sampling process. Future work will focus on incremental sampling, where the model only processes newly added words. This would make the system more viable for real-time email analysis and larger-scale social network monitoring.
Summary updated