Real-time intelligent big data processing: technology, platform, and applications

Tongya Zheng, Gang Chen, Xinyu Wang, Chun Chen, Xingen Wang, Sihui Luo

TL;DR

This paper introduces Stream Cube, an innovative incremental processing technology designed to bridge the gap between slow batch processing and memory-constrained stream processing. By integrating this engine into a "dual-core" intelligent system, the authors demonstrate how to perform complex AI-driven risk detection and behavior analysis on massive data streams in real-time.

What Problem Does This Paper Address?

In the modern "ternary CPH space" (Cyber, Physical, and Human society), we are drowning in data. Traditionally, engineers have had to choose between two suboptimal paths for big data:

  1. Batch Processing (e.g., Hadoop, Spark): Excellent for processing billions of historical records but suffers from high latency (hours or days), making it useless for immediate decision-making.
  2. Stream Processing (e.g., Flink, Storm): Great for low-latency tasks but often struggles with large data volumes, historical context, or changing analysis patterns due to memory constraints.

Furthermore, most existing big data platforms lack native, high-performance integration with Artificial Intelligence. The authors aim to solve this "timeliness vs. volume" trade-off while providing a platform that can deploy AI models (like GBDT or Deep Neural Networks) directly onto live data streams.

Methods

The researchers propose a system architecture centered around two "cores": the Stream Cube engine and an AI Model module.

  • Stream Cube & Incremental Computation: Instead of re-calculating metrics from scratch, Stream Cube uses an incremental scheme. It decomposes complex statistical metrics (like variance or K-order central moments) into mid-term representations using polynomial decomposition. This allows the system to update results instantly as new data arrives without scanning historical logs.
  • Redundancy Elimination: The system identifies overlapping time intervals between different queries. If one user wants a 5-hour window and another wants a 4-hour window, Stream Cube shares the underlying computation to save resources.
  • Dual-Core Intelligence: The system uses PMML (Predictive Model Markup Language) to provide a unified interface. This allows models trained offline on historical "cold" data to be seamlessly "plugged in" to the real-time "hot" data stream for instant inference.

Datasets

The authors evaluated their system using both simulated and real-world application contexts:

  • Simulated Transaction Data: A dataset of 15 million records spanning 150 days. It mimics real-world distributions (Gaussian) for transaction times, amounts (up to 1M CNY), and locations (500 cities).
  • Real-World Scenarios: The paper discusses the data demands of China UMS (50,000 transactions/sec), Taobao (220,000 transactions/sec during peak sales), Shanghai Railway (200,000 IoT records/sec), and the 12306 Ticketing System (1.7 million page views/sec).

Key Insights and Ideas

  • The Value of Recency: The authors argue that data produced ten years ago is far less valuable than data produced in the last hour. Real-time data is the "hottest" and most actionable.
  • Breaking the Memory Wall: A major "aha moment" in their comparison with Apache Flink showed that while Flink is excellent for short windows, its memory usage explodes as the time window grows (e.g., 90 days). Stream Cube’s incremental approach keeps computation time nearly flat regardless of the window size because it only processes the change in data.
  • Beyond Simple Triggers: Most stream systems handle simple "if-this-then-that" logic. By using polynomial decomposition, Stream Cube can handle high-level math (covariance, standard deviation) on the fly, which is essential for sophisticated AI features.

Key Results

  • Throughput and Scaling: On a single node, Stream Cube achieved over 80,000 write TPS and 400,000 read TPS. When scaled to 8 nodes, it reached 2.17 million read TPS, demonstrating near-linear scalability.
  • Latency: Average operation latency remained between 1 and 2 milliseconds, even under high concurrency.
  • Stream Cube vs. Flink: In a 90-day window test, Flink exhausted available memory (128GB) and failed to complete the task, while Stream Cube finished in roughly 150 seconds with stable CPU utility (~90%).
  • Real-World Impact: In production at China UMS, the system responds to risk management queries in less than 50 milliseconds while processing over 50,000 transactions per second.

Limitations and Future Work

While the system is powerful, the authors acknowledge several ongoing challenges:

  • The "Cat and Mouse" Game: In the 12306 railway ticketing scenario, distinguishing between sophisticated bots (using crowdsourced CAPTCHA solving) and humans remains an evolving struggle.
  • Data Noise: Real-world data from third parties is often "noisy." The authors note that more robust automated data cleaning and integration tools are needed.
  • Next-Gen Inference: The paper points toward the "real-time hypergraph" as the next frontier—moving beyond simple feature vectors to complex, evolving relationship networks for even deeper intelligence.

Summary updated

Abstract

Human beings keep exploring the physical space using information means. Only recently, with the rapid development of information technologies and the increasing accumulation of data, human beings can learn more about the unknown world with data-driven methods. Given data timeliness, there is a growing awareness of the importance of real-time data. There are two categories of technologies accounting for data processing: batching big data and streaming processing, which have not been integrated well. Thus, we propose an innovative incremental processing technology named after Stream Cube to process both big data and stream data. Also, we implement a real-time intelligent data processing system, which is based on real-time acquisition, real-time processing, real-time analysis, and real-time decision-making. The real-time intelligent data processing technology system is equipped with a batching big data platform, data analysis tools, and machine learning models. Based on our applications and analysis, the real-time intelligent data processing system is a crucial solution to the problems of the national society and economy.
BibTeX
@article{zheng2019real,
  author = {Tongya Zheng and Gang Chen and Xinyu Wang and Chun Chen and Xingen Wang and Sihui Luo},
  title = {{Real-time intelligent big data processing: technology, platform, and applications}},
  journal = {Science China Information Sciences},
  year = {2019},
  url = {https://openreview.net/forum?id=n0PcTC-v0fE},
}
Figure