Interpreting the Syntactic and Social Elements of the Tweet Representations via Elementary Property Prediction Tasks

Ganesh J, Manish Gupta, Vasudeva VarmaarXiv 2016 · arXiv

Abstract

Tweet embeddings are often treated as black boxes, making it difficult to understand what linguistic or social information they truly capture. This paper evaluates 13 different tweet representation models across eight elementary property prediction tasks. Our findings reveal that while recurrent models are superior for structural properties like length, simple bag-of-means models can capture word order through statistical correlations. Furthermore, models trained on sentence context, such as Skip-Thought Vectors, are particularly effective at encoding social conversational features.

Introduction

In Natural Language Processing (NLP), dense vector representations (embeddings) are the standard for processing text. However, for the messy and social nature of tweets, it is unclear which "elementary properties"—such as word order, slang usage, or conversational status—are preserved in these vectors. We investigate this by defining a set of probing tasks to determine if a simple classifier can extract specific properties from a tweet's embedding.

Probing Tasks and Methodology

We propose eight tasks divided into syntactic and social categories. These tasks are designed to be "elementary," meaning they test basic properties rather than complex downstream objectives like sentiment.

Table 1: Definitions and objectives of the eight elementary property prediction tasks.

We evaluate 13 models, ranging from traditional unsupervised methods like Bag-of-Words (BOW) and LDA to modern supervised deep learning architectures like CNNs and LSTMs. The study utilized large-scale resources, including 2 million tweets for Tweet2Vec training and GloVe embeddings trained on 2 billion tweets for the Bag of Means (BOM) model. Task-specific data included the Noisy Text Challenge dataset for slang identification and the Ritter et al. dataset for tweet-specific Named Entity Recognition.

Results and Analysis

The experimental results (Table 2) highlight the varying strengths of different embedding techniques.

Table 2: Performance (F1-scores) of unsupervised and supervised models across eight probing tasks.

Structural and Syntactic Insights

  • Length and Recurrence: Recurrent models (LSTM, BLSTM, STV) dominate the "Length" task. The LSTM's near-perfect score (99.79%) suggests that the sequential nature of the architecture naturally tracks the number of tokens processed.
  • The Word Order Paradox: Interestingly, the Bag of Means (BOM) model, which theoretically discards order by averaging word vectors, achieved the highest score on the "Word Order" task (61.25%). This indicates that word positions in tweets are often highly correlated with specific vocabulary, allowing non-sequential models to "cheat" the task via statistical patterns.
  • Slang Detection: The Bi-directional LSTM (BLSTM) outperformed the standard LSTM on slang identification (80.52%), demonstrating that both preceding and succeeding context are crucial for disambiguating non-standard language.

Social Context Insights

  • Conversational Features: Skip-Thought Vectors (STV) excelled at identifying if a tweet was a reply (96.41%). This success is attributed to STV’s training objective of predicting adjacent sentences, which naturally captures the flow of conversation.
  • Temporal Dynamics: Predicting "Reply Time" was the most challenging task for all models, with a peak F1-score of only 35.98%. This suggests that textual content alone is insufficient to capture the temporal aspects of social interaction.

Conclusion

This study provides a diagnostic framework for understanding tweet representations. Our results show that different architectures encode distinct types of information: recurrent models capture structure, while context-predictive models capture social dynamics. These insights can guide the development of more specialized models for social media analysis, such as conversation-aware architectures.

Summary updated

Abstract

Research in social media analysis is experiencing a recent surge with a large number of works applying representation learning models to solve high-level syntactico-semantic tasks such as sentiment analysis, semantic textual similarity computation, hashtag prediction and so on. Although the performance of the representation learning models are better than the traditional baselines for the tasks, little is known about the core properties of a tweet encoded within the representations. Understanding these core properties would empower us in making generalizable conclusions about the quality of representations. Our work presented here constitutes the first step in opening the black-box of vector embedding for social media posts, with emphasis on tweets in particular. In order to understand the core properties encoded in a tweet representation, we evaluate the representations to estimate the extent to which it can model each of those properties such as tweet length, presence of words, hashtags, mentions, capitalization, and so on. This is done with the help of multiple classifiers which take the representation as input. Essentially, each classifier evaluates one of the syntactic or social properties which are arguably salient for a tweet. This is also the first holistic study on extensively analysing the ability to encode these properties for a wide variety of tweet representation models including the traditional unsupervised methods (BOW, LDA), unsupervised representation learning methods (Siamese CBOW, Tweet2Vec) as well as supervised methods (CNN, BLSTM).
BibTeX
@misc{j2016interpreting,
  author = {Ganesh J and Manish Gupta and Vasudeva Varma},
  title = {{Interpreting the Syntactic and Social Elements of the Tweet Representations via Elementary Property Prediction Tasks}},
  year = {2016},
  eprint = {1611.04887},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.04887},
}
Figure