SCALAR - A Platform for Real-time Machine Learning Competitions on Data Streams

Nedeljko Radulovic, Dihia Boulegane, Albert BifetJ. Open Source Softw 2020 · DOI · DBLP

Introduction

While platforms like Kaggle have revolutionized machine learning by hosting competitions on static datasets, many real-world applications—such as high-frequency trading, IoT monitoring, and network security—rely on continuous data streams. In these environments, models must process instances incrementally, inspect them only once, and adapt to "concept drift" (changes in data distribution) in real-time. SCALAR is an open-source platform designed to fill this gap, providing the infrastructure to host "Kaggle-style" competitions specifically for streaming data.

System Architecture and Methods

SCALAR is built on a modern, microservices-oriented tech stack optimized for high throughput and low latency. The architecture is divided into three primary layers: Data Sources, the Streaming Engine, and the Application layer.

The SCALAR system architecture, highlighting the integration of Apache Kafka, Spark, and gRPC for real-time data processing and evaluation.
  • Streaming Engine: Apache Kafka handles bi-directional communication, streaming data to participants and receiving predictions in real-time.
  • Evaluation: An online evaluation engine powered by Apache Spark and Kafka calculates performance metrics on the fly, maintaining a live leaderboard.
  • Communication: The platform uses gRPC and Protocol Buffers (Protobuf) for efficient, language-agnostic communication, allowing participants to use various programming languages (e.g., Python, Java, R).
  • Deployment: The environment is containerized using Docker to ensure consistency across different hosting environments.

The Streaming Learning Paradigm

Unlike batch learning, where a model is trained on a fixed set and then tested, SCALAR enforces a "test-then-train" (prequential evaluation) workflow. This ensures that models are evaluated on their ability to predict unseen data before they are allowed to learn from the ground truth.

The prequential evaluation workflow where the model transitions between Ready, Test (prediction), and Train (update) states as it receives data.

In this cycle, the platform sends unlabeled data to the participant's model. The model submits a prediction, and only after the prediction is recorded does the platform release the true label. The model then uses this label to update itself incrementally before the next data instance arrives.

Data Handling and Real-world Application

SCALAR consumes standard CSV files and transforms them into continuous streams based on configurable parameters, such as time intervals and batch sizes. To address the "cold start" problem—where a model has no prior information—the platform uses a two-phase batching approach: an Initial Batch containing both features and targets for baseline training, followed by Regular Batches of streaming data.

The platform's robustness was demonstrated at the IEEE Big Data 2019 Cup Challenges, where it successfully hosted the "Real-time Machine Learning Competition on Data Streams." The deployment proved that the architecture could handle multiple concurrent users and maintain a live leaderboard without significant latency.

Conclusion

SCALAR provides the necessary "plumbing" for the machine learning community to benchmark online learning algorithms in a standardized, competitive environment. By open-sourcing the platform, the authors have enabled researchers to move beyond static datasets and tackle the complexities of real-time data processing.

Summary updated

BibTeX
@article{radulovic2020scalar,
  author = {Nedeljko Radulovic and Dihia Boulegane and Albert Bifet and Nedeljko Radulovic and Dihia Boulegane},
  title = {{SCALAR - A Platform for Real-time Machine Learning Competitions on Data Streams}},
  journal = {J. Open Source Softw.},
  year = {2020},
  doi = {10.21105/joss.02676},
}
Figure