Introduction
Graph Machine Learning (GML) has seen significant advancements, yet the community continues to rely heavily on legacy datasets such as Cora, CiteSeer, and PubMed. While these citation networks are foundational, they often fail to represent the complexities of modern social media environments. Specifically, existing benchmarks are frequently multi-class and based on undirected relationships, which may not align with researchers focusing on binary classification or directed social interactions.
To address this gap, we introduce the Binary Classification Social Network Dataset (BiSND). BiSND is a modern, Twitter-based benchmark designed specifically for binary classification tasks—predicting whether a user account remains active or has been deleted.
The BiSND Dataset
The primary contribution of this work is a curated dataset derived from real-world social media interactions. BiSND consists of:
- Nodes: 12,788 unique Twitter users.
- Edges: 430 directed edges based on user "mentions."
- Features: 19 metadata attributes per node, including follower counts and account age.
- Target: A binary label indicating account status (Existing vs. Deleted).
Compared to traditional citation networks, BiSND is characterized by its high sparsity and directed nature, reflecting the "noisy" and often one-way communication patterns typical of social platforms.
Methodology
The authors developed a rigorous pipeline to extract, annotate, and validate the dataset.
The process began with metadata from 1.29 million users, filtered down to a high-quality set of 12,788. Three distinct graph representations were constructed to evaluate the impact of network structure:
- Node-only: A purely tabular representation ignoring connections.
- Undirected: Connections established if any mention occurred between two users.
- Directed: Connections preserving the original flow of the mention.
A "benchmark gauntlet" was then applied, testing the dataset against classical machine learning (Random Forest, XGBoost), deep learning (MLP), standard Graph Neural Networks (GCN), and state-of-the-art Graph Contrastive Learning (GCL) methods like DAENS and GRACE.
Experimental Results
The evaluation focused on F1-scores to provide a balanced view of binary classification performance.
Key findings from the experiments include:
- Superiority of Self-Supervision: Despite the dataset's sparsity, Graph Contrastive Learning (GCL) models outperformed supervised GNNs. The DAENS model achieved the highest overall F1-score of 70.15%.
- The Importance of Direction: Directed graph representations consistently yielded better results than undirected ones. For instance, a directed GCN achieved an F1 of 67.25%, compared to 66.47% for its undirected version.
- Graph vs. Tabular Data: While traditional models like Random Forest performed strongly on metadata alone (68.73% F1), the top-performing GCL models demonstrated that the underlying social network structure provides a measurable performance boost that tabular data cannot capture.
Discussion and Future Work
BiSND highlights the unique challenges of social network analysis, particularly the issue of sparsity. With only 430 edges for over 12,000 nodes, many users exist as isolated components. However, the fact that GML models still extracted value from this sparse structure is a testament to the importance of relational data.
Future work will focus on enriching the graph density by incorporating other interaction types, such as "follows" and "retweets," and integrating natural language processing (NLP) to extract features directly from tweet content. BiSND serves as a starting point for more nuanced, binary-focused GML research in the social media domain.
Summary updated