Detecting trading trends in financial tick data: the DEBS 2022 grand challenge

Sebastian Frischbier, Jawad Tahir, Christoph Doblander, Arne Hormann, Ruben Mayer, Hans-Arno Jacobsen

Introduction

The financial industry operates on a foundation of "tick data"—a continuous stream of every price change, bid, and ask across global markets. As digital trading has evolved, the volume of this data has reached unprecedented levels. Infront Financial Technology, a major data provider, reported a surge from 18 billion daily events in 2019 to over 40 billion in 2021. The 2022 DEBS (Distributed and Event-Based Systems) Grand Challenge was designed to address the engineering hurdles associated with processing this massive influx of information in real-time.

The core objective of the challenge was to build high-performance, distributed systems capable of identifying "breakout" patterns across thousands of financial instruments simultaneously. This requires a delicate balance between extreme low latency and architectural scalability.

Problem Definition: Complex Event Processing at Scale

Traders rely on technical indicators to make split-second decisions. A common strategy involves identifying "breakout patterns," which occur when short-term price trends cross over long-term trends. While calculating these indicators for a single stock is computationally trivial, performing these operations for over 5,500 instruments in real-time—while managing market-open load spikes and dynamic user interests—presents a significant Complex Event Processing (CEP) challenge.

Methodology and Computational Tasks

The challenge was structured around two primary queries that participants' systems had to execute:

Query 1: Quantitative Indicators

Participants were required to calculate two Exponential Moving Averages (EMA) for every symbol. These calculations utilized 5-minute "tumbling" (non-overlapping) windows with two distinct smoothing factors: $j=38$ and $j=100$.

Query 2: Breakout Pattern Detection

Building on the results of Query 1, the systems had to detect specific crossover events:

  • Bullish Breakouts (Golden Cross): Occurs when the shorter-term EMA (38) overtakes the longer-term EMA (100) from below.
  • Bearish Breakouts (Death Cross): Occurs when the shorter-term EMA (38) falls below the longer-term EMA (100) from above.

The Dynamic Subscription Twist

To mirror real-world trading environments, the evaluation platform, known as Challenger, dynamically modified "subscriptions" during the stream. While the system had to maintain the state for all 5,504 symbols, it was only required to output results for the specific subset of instruments a "trader" was currently monitoring. This forced systems to maintain continuous, accurate state management even for unsubscribed symbols to ensure immediate data availability upon a new subscription.

The "Trading Data" Benchmark Dataset

A significant contribution of the 2022 challenge was the release of a new, publicly available benchmark dataset. The data encompasses 289 million tick events from 5,504 equities and indices across the Paris, Amsterdam, and Frankfurt exchanges, recorded over a full week in November 2021.

For the competition, a "Purged" subset of 59 million price-relevant events was used. The dataset is characterized by a "long-tail" distribution, where a small number of highly active symbols account for the majority of the traffic, and significant load spikes occur during market opening and closing hours.

Architectural Requirements and Insights

The 2022 challenge introduced mandatory Non-functional Requirements (NFRs). Participants were discouraged from creating custom, single-node "hacks" and were instead required to use scalable, industry-standard frameworks. This shift emphasized several key engineering insights:

  • State Management: Systems must calculate EMAs for all symbols continuously. Because the 5-minute windows are "tumbling," the system must handle event-time processing and late-arriving data effectively.
  • Horizontal Scalability: By mandating frameworks like Apache Flink, Kafka, or Spark, the challenge ensured that solutions were capable of being deployed in production data centers rather than remaining purely academic exercises.

Results and Evaluation

The competition saw 17 registered teams, with 8 successfully submitting final solutions. For the first time in the history of the DEBS Grand Challenge, every final submission utilized distributed open-source frameworks such as Flink, Kafka, and Spark.

Solutions were benchmarked using the Challenger platform, which provided a remote and reproducible environment. Ranking was determined by a combination of throughput and 90th percentile latency, ensuring that the winning systems were both fast and consistent under load.

Conclusion and Future Work

The DEBS 2022 Grand Challenge successfully pushed the boundaries of real-time financial data processing. While the challenge focused on relatively simple EMA crossovers, it established a robust framework for evaluating CEP systems. Future iterations may introduce more complex technical indicators, bid/ask volume analysis, or machine learning-based trend prediction. The release of the granular "Trading Data" dataset remains a valuable resource for the research community to continue exploring high-volume event processing.

Summary updated

BibTeX
@inproceedings{frischbier2022detecting,
  author = {Sebastian Frischbier and Jawad Tahir and Christoph Doblander and Arne Hormann and Ruben Mayer and Hans-Arno Jacobsen},
  title = {{Detecting trading trends in financial tick data: the DEBS 2022 grand challenge}},
  booktitle = {DEBS},
  year = {2022},
  doi = {10.1145/3524860.3539645},
}
Figure