Dynamic Data Capture from Social Media Streams: A Contextual Bandit Approach

Thibault Gisselbrecht, Sylvain Lamprier, Patrick Gallinari

Abstract

Social media platforms provide streaming APIs that allow for the dynamic capture of user activity. However, maximizing the utility of collected data under strict API constraints—such as limits on the number of users one can follow—is a complex task. This is further complicated by the non-stationary nature of user behavior, where a relevant source today may become irrelevant tomorrow. We propose a contextual bandit approach that anticipates which profiles are likely to publish relevant content and dynamically updates the set of followed accounts. Our method accounts for both API restrictions and the "hidden context" of unobserved users. Experiments on Twitter datasets and a live deployment demonstrate that our approach significantly outperforms static and stationary baselines.

The Challenge of Social Media Streaming

Tracking specific topics on platforms like Twitter (e.g., "politics" or "science") is hindered by significant API bottlenecks. While millions of users may be active, Twitter’s "Follow" API typically restricts researchers to tracking approximately 5,000 accounts simultaneously.

The traditional approach involves selecting a static "bucket" of relevant accounts. However, social media is inherently non-stationary. A user’s relevance fluctuates; a political commentator might pivot to personal content, while a previously obscure user might suddenly become a primary source for breaking news. To maximize data utility, a system must solve three problems:

  1. Selection: Identifying the best subset of users from a massive pool.
  2. Dynamics: Adapting the list in real-time as behaviors shift.
  3. Information Gap: Managing "hidden contexts," where the activity of users not currently followed is invisible, making it difficult to decide when to switch to them.

A Contextual Bandit Framework

We formalize this task as a Contextual Multi-Armed Bandit (MAB) problem with multiple action selections. In this framework, choosing to follow a user is an action, and the relevance of their tweets constitutes the reward. The "context" is derived from the user's recent activity.

Overview of the iterative dynamic data capture process using a contextual bandit framework.

Standard contextual bandit algorithms assume all contexts are visible before a choice is made. To address the hidden context of unobserved users, we utilize a hybrid model:

  • Global Trends ($\beta$): A shared weight vector that learns how specific features (e.g., topics) correlate with rewards across the entire network.
  • User Bias ($\theta_a$): An individual "intrinsic quality" score for each user.
  • Enhanced Exploration: For users whose current activity is hidden, the algorithm uses an empirical mean of past behavior combined with an exploration term that increases based on the time since the user was last followed. This ensures the system periodically "re-checks" potentially relevant sources.

Experimental Evaluation

The approach was evaluated across three distinct scenarios:

  1. USElections: 2.1 million messages from 5,000 users during the 2012 US presidential election.
  2. Libya: 1.2 million messages from 17,341 users over three months.
  3. Live Twitter Experiment: A two-week real-time run comparing the algorithm against a static list of 5,000 high-quality "expert" accounts.

Key Results

  • Superior Accumulation: In offline tests, the contextual approach consistently accumulated higher cumulative rewards (more relevant tweets) than stationary bandit algorithms like CUCB.
  • Outperforming Experts: In the live experiment, the algorithm initially lagged behind the curated expert list while it learned the environment. However, by iteration 80, it overtook the static list, successfully identifying "rising stars" and dropping accounts that had gone silent or off-topic.
  • Efficiency under Partial Information: Even when context was available for only 10% of users, the algorithm leveraged partial information to outperform non-contextual methods.

Conclusion

By framing social media data collection as a dynamic learning task, we can overcome the limitations of static API filters. The contextual bandit approach allows for a self-correcting system that adapts to the shifting landscape of social media discourse, ensuring that data collection remains relevant even as user behaviors evolve. Future work will focus on improving theoretical regret bounds for hidden contexts and reducing the computational complexity of high-dimensional feature sets.

Summary updated

Abstract

Social Media usually provide streaming data access that enable dynamic capture of the social activity of their users. Leveraging such APIs for collecting social data that satisfy a given pre-defined need may constitute a complex task, that implies careful stream selections. With user-centered streams, it indeed comes down to the problem of choosing which users to follow in order to maximize the utility of the collected data w.r.t. the need. On large social media, this represents a very challenging task due to the huge number of potential targets and restricted access to the data. Because of the intrinsic non-stationarity of user's behavior, a relevant target today might be irrelevant tomorrow, which represents a major difficulty to apprehend. In this paper, we propose a new approach that anticipates which profiles are likely to publish relevant contents - given a predefined need - in the future, and dynamically selects a subset of accounts to follow at each iteration. Our method has the advantage to take into account both API restrictions and the dynamics of users' behaviors. We formalize the task as a contextual bandit problem with multiple actions selection. We finally conduct experiments on Twitter, which demonstrate the empirical effectiveness of our approach in real-world settings.
BibTeX
@inproceedings{gisselbrecht2021dynamic,
  author = {Thibault Gisselbrecht and Sylvain Lamprier and Patrick Gallinari},
  title = {{Dynamic Data Capture from Social Media Streams: A Contextual Bandit Approach}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v10i1.14734},
}
Figure