SIEVE: Helping developers sift wheat from chaff via cross-platform analysis

Agus Sulistya, Gede Artha Azriadi Prana, Abhishek Sharma, David Lo, Christoph Treude

Introduction

Software developers increasingly rely on a vast ecosystem of social media and community forums beyond official documentation and IDEs. However, the "signal-to-noise" ratio varies drastically across these platforms. While specialized forums like Stack Overflow are highly moderated and technical, general-purpose platforms like Twitter and YouTube are often "noisy," filled with personal updates, memes, or non-technical chatter. This information overload makes it difficult for automated tools to extract actionable knowledge.

To address this, researchers developed SIEVE, a transfer learning framework designed to leverage high-quality technical data from "rich" platforms to filter through the noise of "open" platforms.

The SIEVE Methodology

SIEVE utilizes a two-stage approach based on transfer representation learning. The core idea is that the mathematical understanding of technical concepts learned from a clean environment can be transferred to a messy one.

The two-stage SIEVE framework for transfer representation learning.

Stage 1: Representation Learning

The framework first builds domain-specific word embedding models using Word2Vec's skip-gram architecture. These models are trained on massive datasets from Stack Overflow and the Software Engineering Stack Exchange. By mapping technical terms into a high-dimensional space, the model ensures that semantically related words—such as "Python" and "Django"—are positioned close together, capturing the specific nuances of software engineering vocabulary.

Stage 2: Task Resolution

The pre-trained embeddings are then applied to specific "target" platforms:

  • Twitter Ranking: To identify software-related tweets, SIEVE calculates the cosine similarity between a tweet's vector and "gold standard" technical sentences sampled from the source platforms.
  • YouTube Classification: To identify informative comments on coding tutorials, the embeddings serve as features for a Support Vector Machine (SVM) classifier.

Leveraging Structured Data

The success of SIEVE relies on the inherent structure of its source platforms. Unlike a 280-character tweet, a Stack Overflow post contains multiple layers of metadata, including titles, tags, and community-voted answers, which provide a rich context for training word embeddings.

Anatomy of a Stack Overflow post, illustrating the high-quality structured data used for training.

Key Insights

Domain Specificity vs. Corpus Size

A major finding of the study was that SIEVE outperformed models trained on the Google News dataset. Despite Google News being significantly larger (100 billion words), its embeddings are general. In software engineering, words like "string," "float," or "tree" have highly specific meanings that general-purpose models fail to capture. SIEVE’s domain-specific training allows it to recognize these technical contexts accurately.

Cross-Platform Synergy

The research demonstrates that knowledge is not siloed. The way developers discuss problems on Stack Overflow is similar enough to how they share tips on Twitter that the linguistic patterns of one can be directly applied to the other. This synergy allows SIEVE to capture the intent and context of a sentence rather than relying on simple, repetitive keywords.

Performance Results

SIEVE demonstrated significant improvements over previous state-of-the-art baselines:

  • Twitter: The framework achieved an Accuracy@K of 0.800 to 0.980 for identifying software-related tweets, outperforming the NIRMAL baseline.
  • YouTube: For classifying informative comments, SIEVE reached an F-measure of 0.873, a 10.3% improvement over traditional term-frequency approaches.
  • Consistency: Models trained on the broader Software Engineering Stack Exchange (SIEVE_SE) performed slightly better than those trained strictly on Stack Overflow (SIEVE_SO), likely due to the more diverse, high-level architectural discussions found on the broader exchange.

Conclusion and Future Work

SIEVE proves that transfer learning is a powerful tool for navigating the noise of social media in technical domains. While the current model uses specific heuristics—such as limiting source sentences to 140 characters to match Twitter—future iterations aim to incorporate even broader data sources like GitHub README files, technical blogs, and Wikipedia to create a more robust representation of software engineering knowledge.

Summary updated

Abstract

Software developers have benefited from various sources of knowledge such as forums, question-and-answer sites, and social media platforms to help them in various tasks. Extracting software-related knowledge from different platforms involves many challenges. In this paper, we propose an approach to improve the effectiveness of knowledge extraction tasks by performing cross-platform analysis. Our approach is based on transfer representation learning and word embedding, leveraging information extracted from a source platform which contains rich domain-related content. The information extracted is then used to solve tasks in another platform (considered as target platform) with less domain-related content. We first build a word embedding model as a representation learned from the source platform, and use the model to improve the performance of knowledge extraction tasks in the target platform. We experiment with Software Engineering Stack Exchange and Stack Overflow as source platforms, and two different target platforms, i.e., Twitter and YouTube. Our experiments show that our approach improves performance of existing work for the tasks of identifying software-related tweets and helpful YouTube comments.
BibTeX
@article{sulistya2020sieve,
  author = {Agus Sulistya and Gede Artha Azriadi Prana and Abhishek Sharma and David Lo and Christoph Treude},
  title = {{SIEVE: Helping developers sift wheat from chaff via cross-platform analysis}},
  journal = {Empir. Softw. Eng.},
  year = {2020},
  doi = {10.1007/s10664-019-09775-w},
}
Figure