Abstract
Effective semantic sentence embeddings typically require large-scale, human-annotated datasets which are costly and often limited to formal language. This study explores the use of Twitter's intrinsic interaction signals—specifically quotes and replies—as a source of "weakly similar" pairs for training. By fine-tuning Transformer-based models on these informal interactions, we demonstrate significant performance gains on social media similarity tasks while remaining competitive on formal benchmarks. Our findings suggest that the vast diversity of human conversation on social media can provide sufficient signal for learning robust semantic representations without manual labeling.
Introduction
Sentence embeddings are critical for tasks like semantic search and paraphrase identification. However, standard training sets like the Semantic Textual Similarity benchmark (STSb) are predominantly composed of formal text (e.g., news, Wikipedia). This creates a domain gap when models are applied to the noisy, slang-heavy environment of social media. This paper addresses this gap by exploiting Twitter's structural interactions to find related sentences automatically, bypassing the need for human annotation.
Methodology
The authors evaluate several Transformer architectures, including RoBERTa, S-BERT, and BERTweet. To optimize the embeddings, they compared two loss functions: Triplet Loss and Multiple Negative Loss (MNLoss). MNLoss proved significantly more efficient and effective by treating all other positive pairs in a training batch as negatives for a given anchor. Furthermore, MEAN pooling of tokens from the final layer was found to outperform standard [CLS] token representations.
Datasets and Evaluation
Four datasets were derived from a collection of 75 million tweets:
- Quote (Qt): A tweet and its quoted counterpart.
- Reply (Rp): A tweet and its direct reply.
- Co-quote (CoQt): Two tweets quoting the same source.
- Co-reply (CoRp): Two replies to the same source.
Models were evaluated on established benchmarks (PIT-2015, TURL, STSb) and four novel ranking tasks (DQ, DR, CQ, CR) designed to test the model's ability to identify related tweets from a pool of candidates.
Results and Analysis
The experimental results highlight the superiority of domain-specific pre-training and the effectiveness of weak supervision.
As shown in Table 1, the "Our-BERTweet (all)" model achieved an average nDCG of 77.1 on the novel Twitter benchmarks, nearly doubling the performance of the base BERTweet model (47.5). On the PIT-2015 paraphrase task, the proposed model reached a Pearson’s $r$ of 66.1, a substantial improvement over the 43.8 achieved by standard S-BERT.
Ablation studies (Figure 1) reveal that performance does not plateau even at 2 million samples, suggesting that larger corpora would likely continue to improve embedding quality. Additionally, the sensitivity to batch size confirms the importance of the MNLoss approach.
Discussion and Limitations
A key insight is that "weak" supervision—where pairs are related but not necessarily paraphrases—is sufficient for learning semantics. While the models were trained on noisy data, they remained competitive on the formal STSb benchmark (scoring 72.4), indicating that the learned semantics are generalizable.
Limitations include potential temporal bias, as the data was collected over a short window in 2020. Future work should explore multilingual applications and the performance of these embeddings on downstream tasks like sentiment analysis.
Conclusion
Twitter interactions provide a rich, scalable, and cost-effective resource for training semantic sentence embeddings. By leveraging domain-specific models and efficient loss functions, researchers can bridge the gap between formal and informal text processing.
Summary updated