Abstract
Tweet embeddings are often treated as black boxes, making it difficult to understand what linguistic or social information they truly capture. This paper evaluates 13 different tweet representation models across eight elementary property prediction tasks. Our findings reveal that while recurrent models are superior for structural properties like length, simple bag-of-means models can capture word order through statistical correlations. Furthermore, models trained on sentence context, such as Skip-Thought Vectors, are particularly effective at encoding social conversational features.
Introduction
In Natural Language Processing (NLP), dense vector representations (embeddings) are the standard for processing text. However, for the messy and social nature of tweets, it is unclear which "elementary properties"—such as word order, slang usage, or conversational status—are preserved in these vectors. We investigate this by defining a set of probing tasks to determine if a simple classifier can extract specific properties from a tweet's embedding.
Probing Tasks and Methodology
We propose eight tasks divided into syntactic and social categories. These tasks are designed to be "elementary," meaning they test basic properties rather than complex downstream objectives like sentiment.
We evaluate 13 models, ranging from traditional unsupervised methods like Bag-of-Words (BOW) and LDA to modern supervised deep learning architectures like CNNs and LSTMs. The study utilized large-scale resources, including 2 million tweets for Tweet2Vec training and GloVe embeddings trained on 2 billion tweets for the Bag of Means (BOM) model. Task-specific data included the Noisy Text Challenge dataset for slang identification and the Ritter et al. dataset for tweet-specific Named Entity Recognition.
Results and Analysis
The experimental results (Table 2) highlight the varying strengths of different embedding techniques.
Structural and Syntactic Insights
- Length and Recurrence: Recurrent models (LSTM, BLSTM, STV) dominate the "Length" task. The LSTM's near-perfect score (99.79%) suggests that the sequential nature of the architecture naturally tracks the number of tokens processed.
- The Word Order Paradox: Interestingly, the Bag of Means (BOM) model, which theoretically discards order by averaging word vectors, achieved the highest score on the "Word Order" task (61.25%). This indicates that word positions in tweets are often highly correlated with specific vocabulary, allowing non-sequential models to "cheat" the task via statistical patterns.
- Slang Detection: The Bi-directional LSTM (BLSTM) outperformed the standard LSTM on slang identification (80.52%), demonstrating that both preceding and succeeding context are crucial for disambiguating non-standard language.
Social Context Insights
- Conversational Features: Skip-Thought Vectors (STV) excelled at identifying if a tweet was a reply (96.41%). This success is attributed to STV’s training objective of predicting adjacent sentences, which naturally captures the flow of conversation.
- Temporal Dynamics: Predicting "Reply Time" was the most challenging task for all models, with a peak F1-score of only 35.98%. This suggests that textual content alone is insufficient to capture the temporal aspects of social interaction.
Conclusion
This study provides a diagnostic framework for understanding tweet representations. Our results show that different architectures encode distinct types of information: recurrent models capture structure, while context-predictive models capture social dynamics. These insights can guide the development of more specialized models for social media analysis, such as conversation-aware architectures.
Summary updated