BlogTrackers: A Tool for Sociologists to Track and Analyze Blogosphere

Nitin Agarwal, Shamanth Kumar, Huan Liu, Mark Woodward

TL;DR

BlogTrackers is a specialized tool designed to help sociologists navigate and analyze the vast, noisy landscape of the blogosphere. By combining custom web crawling with advanced analytics—such as influencer identification and "burst" detection—it allows researchers to move beyond manual observation to data-driven social analysis.

Problem Statement

In 2009, the blogosphere was exploding, with services like Technorati indexing over 133 million blog records. For sociologists, this represented a goldmine of socio-behavioral data, but the sheer volume made manual "eyeballing" of sites impossible.

The authors identify a significant gap: existing tools were either too generic (standard search engines) or lacked the specific analytical depth required for social science. Sociologists don't just need to find a keyword; they need to identify influential actors, track how topics evolve over time, and understand the "pulse" of specific communities. BlogTrackers was built to bridge this gap by providing an integrated pipeline for data collection and multi-faceted analysis.

System Architecture and Methodology

BlogTrackers is a Java-based desktop application that utilizes a relational database backend. Its architecture is split into two primary phases:

System architecture of BlogTrackers showing the data collection and analysis pipeline.
  1. Data Collection (The Crawler):
    • Batch Crawler: Uses HTML scraping and regular expressions to pull historical data from blog sites.
  2. RSS Crawler: A more lightweight, scheduled crawler that monitors feeds for real-time updates, ensuring the database stays current without constant full-site scraping.
  3. Analysis (The Tracker):
    • Blog Analysis: Includes a Term Frequency Analyzer (generating tag clouds to visualize dominant themes) and a Traffic Pattern Graph to identify temporal "bursts" in activity.
  4. Blogger Analysis: This is a standout feature. Instead of just counting posts, it uses an algorithm to categorize bloggers into a 2x2 matrix: Active-Influential, Inactive-Influential, Active-Non Influential, and Inactive-Non Influential.
  5. Search & Alerts: Uses the Apache Lucene engine for high-performance indexing and allows users to set up watchlists for specific keywords, triggering alerts when new relevant content is posted.

Data Sources

The tool is designed to be source-agnostic, allowing researchers to point the crawler at any blog community of interest. At the time of writing, it was tracking 10 different data sources, including:

  • Twitter (then in its early high-growth phase)
  • Engadget
  • The Unofficial Apple Weblog (TUAW)
  • LiveJournal

Conceptual Contributions

  • The "Inactive-Influential" Blogger: One of the most interesting conceptual contributions is the recognition that influence isn't always tied to high frequency. A blogger who posts rarely but is cited widely (the "sleeping giant" profile) is often more important to a sociologist than a high-frequency "noise" poster.
  • Drill-Down Capability: The authors emphasize a "drill-down" philosophy. A researcher can start with a high-level traffic burst (e.g., a spike in posts about "iPhone"), click into the tag cloud to see related terms, and then identify the specific influential bloggers driving that conversation.
  • Integration is Key: By combining the crawler and the analyzer into a single tool, the authors reduce the technical barrier for social scientists who might otherwise need to stitch together disparate scripts and databases.

Evaluation and Comparative Analysis

The paper functions primarily as a system demonstration. However, it provides a comparative analysis (Table 1) against contemporary tools like Technorati, BlogPulse, and Google Blog Search.

Comparison of BlogTrackers features with other contemporary tools.
  • Feature Parity: BlogTrackers was shown to be the most comprehensive tool of its time, offering features like influencer identification and automated alerts that were missing from major commercial competitors.
  • Visualizing Trends: The authors demonstrate the tool's ability to map out the "Apple" blogosphere, successfully identifying product launch cycles through traffic bursts and identifying key opinion leaders within that niche.

Limitations and Future Directions

The authors acknowledge several areas for growth:

  • Sentiment Analysis: The 2009 version focused on volume and influence but did not yet programmatically determine the tone (positive/negative) of the discourse.
  • Clustering: They suggest adding automated blog clustering to group similar communities together.
  • Scaling: As a desktop-based application, there are inherent limits to how much of the "entire" blogosphere one researcher can track compared to a distributed cloud-based system.
  • Conversation Tracking: Future iterations aimed to better map the "pingback" and comment threads to see exactly how a story propagates from one blog to another.

Summary updated

Abstract

We present a tool, BlogTrackers, which helps sociologists to track and analyze blogs of particular interests by designing and integrating unique features. We present an overview of BlogTrackers, illustrate its functions of various components of BlogTrackers, and outline future work for expansion in meeting the growing needs of sociologists.
BibTeX
@inproceedings{agarwal2009blogtrackers,
  author = {Nitin Agarwal and Shamanth Kumar and Huan Liu and Mark Woodward},
  title = {{BlogTrackers: A Tool for Sociologists to Track and Analyze Blogosphere}},
  booktitle = {Icwsm},
  year = {2009},
}
Figure