An Empirical Study on Learning to Rank of Tweets

Yajuan Duan, Long Jiang, Tao Qin, Ming Zhou, Heung-Yeung ShumCOLING 2010

TL;DR

This paper tackles the challenge of ranking tweets for relevance beyond simple chronological order or retweet counts. They propose a learning-to-rank approach that combines content, Twitter-specific, and account authority features, finding that tweet URL presence, account authority (specifically, being listed by others), and tweet length are the most impactful features for effective tweet ranking.

What Problem Does This Paper Address?

Twitter's default chronological ordering of tweets, while providing recency, often buries relevant information under a flood of less important or even spam content. While Twitter has introduced a popularity-based ranking (using retweet counts), this also falls short by not considering content relevance or the authority of the tweeting account. This means users searching for specific information, like consumer opinions on a product, might miss crucial tweets if they aren't among the most retweeted or most recent. The paper aims to address this by developing a more sophisticated ranking system that prioritizes content relevance and account influence.

Methods

The core of the paper's methodology is a learning-to-rank (LTR) approach. They frame tweet ranking as a supervised learning problem where the goal is to train a model that can predict the relevance of a tweet to a given query.

Here's a breakdown of their approach:

  • Feature Engineering: They meticulously extract three categories of features:
  • Content Relevance Features: These measure how well the tweet's text matches the search query. They use standard IR metrics like Okapi BM25 and cosine similarity based on TFIDF vectors. They also include tweet length as a proxy for information richness.
  • Twitter-Specific Features: These capture unique aspects of tweets, such as the presence and count of URLs, retweet counts, hashtag scores, and whether a tweet is a reply.
  • Account Authority Features: This is a key contribution. Instead of just using follower counts (which they found to be a poor indicator), they propose and test several metrics derived from user relationships, including follower count, mention count, list count, and importantly, the PageRank-based popularity score calculated from retweet relationships. They also explore "first" and "important" versions of these scores for the tweet's author.
  • Learning Algorithm: They employ the RankSVM algorithm, a well-established method for learning ranking functions.
  • Feature Selection: To identify the most impactful features, they use an "advanced greedy feature selection algorithm" to find the optimal combination of features.

Datasets

  • Training and Evaluation Data: They collected 162,626 English tweets for 20 query terms (5 persons, 5 locations, 5 products, 5 movies) from Twitter Search between March 25, 2010, and April 2, 2010. After deduplication, they had 159,298 tweets.
  • Annotation: For each query, they sampled 500 tweets and had a human editor label them with a relevance grade (Excellent, Good, Fair, Bad). This resulted in a dataset with a significant portion of "bad" (irrelevant) tweets.
  • Benchmark: They compare their proposed method against ranking by chronological order, account authority alone, and content relevance alone.

Key Insights and Ideas

  • Beyond Retweets: The paper strongly argues that retweet count is an insufficient metric for relevance, especially in the face of potential spam.
  • Account Authority Matters: The influence of the tweet's author is a significant factor in determining relevance.
  • "Listed by Others" is Key: Their most impactful finding regarding account authority is that the number of times an account is listed by other users (a proxy for being part of curated lists or recognized by others) is a more effective indicator of authority than raw follower count.
  • URL is King (for relevance): The presence of a URL in a tweet is identified as the single most effective feature for ranking, likely because URLs often link to more detailed, relevant information.
  • Feature Combination is Crucial: While individual features have impact, their optimal combination (URL, Sum_mention, First_List, Length, Important_follower) yields the best performance.

Key Results

  • Their proposed learning-to-rank model, RankSVM_Best, which uses an optimized feature set, significantly outperforms baseline methods.
  • Specifically, RankSVM_Best achieved approximately 15.3% improvement on NDCG@10 compared to RankSVM_Full (which used all features).
  • The URL feature alone, when removed from the best feature combination, caused a serious performance decline (p < 0.001).
  • The "number of lists" feature also showed a significant contribution (p < 0.05) when removed.
  • Their model significantly outperformed ranking by time, account authority, and content relevance individually, with statistical significance (p < 0.01 for individual baselines, p < 0.05 for RankSVM_Full).

Limitations and Future Work

  • Data Sparsity and Size: The authors acknowledge that the dataset size (20 queries) might be insufficient for robust training, and feature selection can be prone to local optima due to data sparseness.
  • Annotation Effort: To minimize annotation effort, they focused on tweets returned by Twitter Search for a given query. This might limit the evaluation of content relevance features, as tweets not containing the query term were excluded.
  • Retweet Detection: Their current method for identifying retweets is basic and might miss nuanced retweet information where the original tweet is modified.
  • Spam Filtering: The current ranking system does not explicitly address spam, which is a critical issue for any search engine.
  • Future Directions:
    • Explore query expansion techniques to improve recall.
  • Develop more sophisticated spam filtering approaches.
  • Use a larger, more diverse dataset, potentially including tweets from the Twitter firehose.
  • Investigate more accurate retweet relation detection methods.

Summary updated

BibTeX
@inproceedings{duan2010empirical,
  author = {Yajuan Duan and Long Jiang and Tao Qin and Ming Zhou and Heung-Yeung Shum},
  title = {{An Empirical Study on Learning to Rank of Tweets}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}

Loading related content...

Figure