A Binary Classification Social Network Dataset for Graph Machine Learning

Adnan Ali, Jinglong Li, Huanhuan Chen, AlMotasem Bellah AlAjlouniarXiv 2025 · DOI · arXiv

Introduction

Graph Machine Learning (GML) has seen significant advancements, yet the community continues to rely heavily on legacy datasets such as Cora, CiteSeer, and PubMed. While these citation networks are foundational, they often fail to represent the complexities of modern social media environments. Specifically, existing benchmarks are frequently multi-class and based on undirected relationships, which may not align with researchers focusing on binary classification or directed social interactions.

To address this gap, we introduce the Binary Classification Social Network Dataset (BiSND). BiSND is a modern, Twitter-based benchmark designed specifically for binary classification tasks—predicting whether a user account remains active or has been deleted.

The BiSND Dataset

The primary contribution of this work is a curated dataset derived from real-world social media interactions. BiSND consists of:

  • Nodes: 12,788 unique Twitter users.
  • Edges: 430 directed edges based on user "mentions."
  • Features: 19 metadata attributes per node, including follower counts and account age.
  • Target: A binary label indicating account status (Existing vs. Deleted).

Compared to traditional citation networks, BiSND is characterized by its high sparsity and directed nature, reflecting the "noisy" and often one-way communication patterns typical of social platforms.

Methodology

The authors developed a rigorous pipeline to extract, annotate, and validate the dataset.

The BiSND dataset creation and validation pipeline, from raw Twitter data to model prediction.

The process began with metadata from 1.29 million users, filtered down to a high-quality set of 12,788. Three distinct graph representations were constructed to evaluate the impact of network structure:

  1. Node-only: A purely tabular representation ignoring connections.
  2. Undirected: Connections established if any mention occurred between two users.
  3. Directed: Connections preserving the original flow of the mention.

A "benchmark gauntlet" was then applied, testing the dataset against classical machine learning (Random Forest, XGBoost), deep learning (MLP), standard Graph Neural Networks (GCN), and state-of-the-art Graph Contrastive Learning (GCL) methods like DAENS and GRACE.

Experimental Results

The evaluation focused on F1-scores to provide a balanced view of binary classification performance.

Comparison of performance metrics (F1-Score, Precision, Recall, and Jaccard Score) across different experimental configurations.

Key findings from the experiments include:

  • Superiority of Self-Supervision: Despite the dataset's sparsity, Graph Contrastive Learning (GCL) models outperformed supervised GNNs. The DAENS model achieved the highest overall F1-score of 70.15%.
  • The Importance of Direction: Directed graph representations consistently yielded better results than undirected ones. For instance, a directed GCN achieved an F1 of 67.25%, compared to 66.47% for its undirected version.
  • Graph vs. Tabular Data: While traditional models like Random Forest performed strongly on metadata alone (68.73% F1), the top-performing GCL models demonstrated that the underlying social network structure provides a measurable performance boost that tabular data cannot capture.

Discussion and Future Work

BiSND highlights the unique challenges of social network analysis, particularly the issue of sparsity. With only 430 edges for over 12,000 nodes, many users exist as isolated components. However, the fact that GML models still extracted value from this sparse structure is a testament to the importance of relational data.

Future work will focus on enriching the graph density by incorporating other interaction types, such as "follows" and "retweets," and integrating natural language processing (NLP) to extract features directly from tweet content. BiSND serves as a starting point for more nuanced, binary-focused GML research in the social media domain.

Summary updated

BibTeX
@misc{ali2025binary,
  author = {Adnan Ali and Jinglong Li and Huanhuan Chen and AlMotasem Bellah AlAjlouni},
  title = {{A Binary Classification Social Network Dataset for Graph Machine Learning}},
  year = {2025},
  eprint = {2503.02397},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.02397},
}
Figure