Introduction
Social media researchers frequently rely on Twitter's free Streaming API to gather data. However, this service is limited to a 1% sample of total global traffic. When a query's volume exceeds this threshold, Twitter employs an opaque sampling algorithm, potentially introducing significant bias. This paper, presented at the 2014 WWW Conference, investigates whether this 1% sample is truly representative of the full "Firehose" and proposes a method to detect bias using only publicly available tools.
Methodology: The "Poor Man's Firehose"
The authors introduce a framework to identify periods of data inaccuracy without the need for the expensive 100% Firehose feed. The core idea is to use Twitter's Sample API—a random 1% of all tweets—as a "gold standard" proxy.
The process involves:
- Validation: Proving the Sample API statistically matches the Firehose.
- Bootstrapping: Resampling the Sample API data 100 times to establish a baseline distribution (mean and standard deviation) for hashtag frequencies.
- Z-Score Normalization: Comparing the Streaming API's hashtag frequency against this bootstrapped distribution. If the Streaming API value falls outside $\pm 3$ standard deviations, it is flagged as statistically biased.
Validation of the Sample API
To ensure the Sample API can serve as a reliable proxy, the authors compared it against GNIP Firehose data. Using Kendall’s $\tau$ to measure rank correlation for top hashtags, they found a very high correlation ($\tau_\beta \approx 0.98$ for the top 10 hashtags).
As shown in Figure 1, the Sample API closely tracks the ground truth provided by the Firehose, confirming its utility as a benchmark for representativeness.
Detecting Bias in Practice
The study applied this method to specific hashtags like "#believemovie." By comparing the Streaming API results to the bootstrapped Sample API distribution, the authors could pinpoint specific times when the data was skewed.
In Figure 6, the blue line represents the Streaming API's normalized occurrence, while the red dashed line and error bars represent the expected distribution from the Sample API. Instances where the blue line deviates significantly from the red distribution indicate periods of sampling bias.
Key Findings
- Bias is Temporal: Bias is not a constant state; it fluctuates based on global traffic spikes. A dataset might be representative at one hour and become heavily biased at another if global volume forces more aggressive sampling.
- Global Consistency: The Streaming API is remarkably consistent across different geographic locations (e.g., USA vs. Austria), suggesting sampling occurs at a central level rather than a regional edge.
- Stability: Identical queries started at slightly different times yield nearly identical tweet sets (Jaccard similarity > 0.99).
Conclusion and Limitations
The Sample API offers a powerful, free "thermometer" for researchers to check the health of their data. However, the method is limited by sparsity. Niche or "long-tail" hashtags may not appear in the 1% Sample API at all, making it impossible to establish a baseline for comparison. Future work may involve more advanced statistical techniques to estimate trends for these rarer events.
Summary updated