Introduction
Social media platforms generate a continuous stream of information, making the identification of "noteworthy events"—such as natural disasters or breaking news—a significant challenge. While numerous models for Social Event Detection (SED) have been developed, the field has historically suffered from fragmentation. Researchers often face incompatible codebases, varying data formats, and inconsistent evaluation metrics, hindering the benchmarking of new models and their deployment in industrial settings.
SocialED is a new open-source Python library designed to bridge this gap. It provides a "one-stop-shop" for SED by integrating 19 state-of-the-art algorithms and 14 diverse datasets into a single, scikit-learn-style API. By standardizing the SED workflow, SocialED facilitates both academic research and real-world application.
Unified API and System Architecture
The library is built on a modern tech stack, including PyTorch, Hugging Face Transformers, and scikit-learn, with a focus on modularity and ease of use. The core of SocialED is its unified API design, which follows a predictable four-step workflow:
preprocess(): Handles data preparation, including tokenization and graph construction.fit(): Trains the selected detection algorithm.detection(): Predicts events on new or test data.evaluate(): Provides standardized performance metrics.
This modular design allows components like graph construction to be decoupled from specific models, enabling researchers to mix and match preprocessing techniques with different architectures.
Comprehensive Algorithms and Datasets
SocialED includes 14 widely-used datasets covering general events (e.g., Event2012, MAVEN), crisis management (e.g., CrisisLexT26, HumAID), and multilingual contexts (e.g., ArabicTwitter). These datasets vary in scale from 3,000 to over 85,000 texts.
The library tracks the evolution of the field, supporting 19 algorithms across various backbones, including Topic Modeling (LDA), Word Embeddings (Word2Vec, GloVe), Pre-trained Language Models (BERT, SBERT), and Graph Neural Networks (KPGNN, GCN).
Software Quality and Production Readiness
Unlike many academic projects, SocialED emphasizes high software engineering standards. It features 99% code coverage, continuous integration via GitHub Actions, and strict adherence to PEP 8 style guides. This ensures the library is reliable and ready for production environments. Furthermore, the library is hardware-agnostic, optimized for both CPU and GPU environments to handle large-scale social media streams efficiently.
Conclusion and Future Directions
While SocialED provides a comprehensive foundation, future work will focus on expanding its scope to more languages and cultures, incorporating automated machine learning (AutoML) for hyperparameter optimization, and supporting real-time streaming data for early warning systems.
Availability: SocialED is publicly available on GitHub and can be installed via PyPI (pip install socialed). Detailed documentation is hosted on Read the Docs.
Summary updated