Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings

Marco Di Giovanni, Marco Brambilla

Abstract

Effective semantic sentence embeddings typically require large-scale, human-annotated datasets which are costly and often limited to formal language. This study explores the use of Twitter's intrinsic interaction signals—specifically quotes and replies—as a source of "weakly similar" pairs for training. By fine-tuning Transformer-based models on these informal interactions, we demonstrate significant performance gains on social media similarity tasks while remaining competitive on formal benchmarks. Our findings suggest that the vast diversity of human conversation on social media can provide sufficient signal for learning robust semantic representations without manual labeling.

Introduction

Sentence embeddings are critical for tasks like semantic search and paraphrase identification. However, standard training sets like the Semantic Textual Similarity benchmark (STSb) are predominantly composed of formal text (e.g., news, Wikipedia). This creates a domain gap when models are applied to the noisy, slang-heavy environment of social media. This paper addresses this gap by exploiting Twitter's structural interactions to find related sentences automatically, bypassing the need for human annotation.

Methodology

The authors evaluate several Transformer architectures, including RoBERTa, S-BERT, and BERTweet. To optimize the embeddings, they compared two loss functions: Triplet Loss and Multiple Negative Loss (MNLoss). MNLoss proved significantly more efficient and effective by treating all other positive pairs in a training batch as negatives for a given anchor. Furthermore, MEAN pooling of tokens from the final layer was found to outperform standard [CLS] token representations.

Datasets and Evaluation

Four datasets were derived from a collection of 75 million tweets:

  • Quote (Qt): A tweet and its quoted counterpart.
  • Reply (Rp): A tweet and its direct reply.
  • Co-quote (CoQt): Two tweets quoting the same source.
  • Co-reply (CoRp): Two replies to the same source.

Models were evaluated on established benchmarks (PIT-2015, TURL, STSb) and four novel ranking tasks (DQ, DR, CQ, CR) designed to test the model's ability to identify related tweets from a pool of candidates.

Results and Analysis

The experimental results highlight the superiority of domain-specific pre-training and the effectiveness of weak supervision.

Table 1: Performance comparison of various models across informal and formal benchmarks. Our-BERTweet (all) shows the strongest overall performance on Twitter-specific tasks.

As shown in Table 1, the "Our-BERTweet (all)" model achieved an average nDCG of 77.1 on the novel Twitter benchmarks, nearly doubling the performance of the base BERTweet model (47.5). On the PIT-2015 paraphrase task, the proposed model reached a Pearson’s $r$ of 66.1, a substantial improvement over the 43.8 achieved by standard S-BERT.

Figure 1: Impact of corpus size (left) and batch size (right) on model performance. Performance continues to scale with data volume and larger batch sizes.

Ablation studies (Figure 1) reveal that performance does not plateau even at 2 million samples, suggesting that larger corpora would likely continue to improve embedding quality. Additionally, the sensitivity to batch size confirms the importance of the MNLoss approach.

Discussion and Limitations

A key insight is that "weak" supervision—where pairs are related but not necessarily paraphrases—is sufficient for learning semantics. While the models were trained on noisy data, they remained competitive on the formal STSb benchmark (scoring 72.4), indicating that the learned semantics are generalizable.

Limitations include potential temporal bias, as the data was collected over a short window in 2020. Future work should explore multilingual applications and the performance of these embeddings on downstream tasks like sentiment analysis.

Conclusion

Twitter interactions provide a rich, scalable, and cost-effective resource for training semantic sentence embeddings. By leveraging domain-specific models and efficient loss functions, researchers can bridge the gap between formal and informal text processing.

Summary updated

Abstract

Semantic sentence embeddings are usually supervisedly built minimizing distances between pairs of embeddings of sentences labelled as semantically similar by annotators. Since big labelled datasets are rare, in particular for non-English languages, and expensive, recent studies focus on unsupervised approaches that require not-paired input sentences. We instead propose a language-independent approach to build large datasets of pairs of informal texts weakly similar, without manual human effort, exploiting Twitter's intrinsic powerful signals of relatedness: replies and quotes of tweets. We use the collected pairs to train a Transformer model with triplet-like structures, and we test the generated embeddings on Twitter NLP similarity tasks (PIT and TURL) and STSb. We also introduce four new sentence ranking evaluation benchmarks of informal texts, carefully extracted from the initial collections of tweets, proving not only that our best model learns classical Semantic Textual Similarity, but also excels on tasks where pairs of sentences are not exact paraphrases. Ablation studies reveal how increasing the corpus size influences positively the results, even at 2M samples, suggesting that bigger collections of Tweets still do not contain redundant information about semantic similarities.
BibTeX
@inproceedings{giovanni2021exploiting,
  author = {Marco Di Giovanni and Marco Brambilla},
  title = {{Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=WB6IgYuZut-},
}
Figure