OpenMLDB: A Real-Time Relational Data Feature Computation System for Online ML

Xuanhe Zhou, Wei Zhou, Liguo Qi, Hao Zhang, Dihao Chen, Bingsheng He, Mian Lu, Guoliang Li, Fan Wu, Yuqiang Chen

TL;DR

This paper introduces OpenMLDB, a novel system designed for efficient and consistent real-time feature computation for online machine learning applications. It addresses the critical challenge of bridging the gap between offline model training and online model serving by providing a unified approach, significantly reducing latency and improving performance compared to existing solutions.

What Problem Does This Paper Address?

Online machine learning applications, such as fraud detection and personalized recommendations, rely heavily on real-time feature computation. Traditionally, this process is split into two distinct phases: offline for model training and online for model serving. This separation often leads to inconsistencies between the features used during training and those available in production, requiring extensive verification. Furthermore, many online ML features involve complex time-series computations and multi-table operations that existing data processing systems (like Spark and Flink) struggle to handle efficiently, leading to multi-second latencies that are unacceptable for real-time applications. The paper aims to solve these issues by providing a unified, high-performance system for both offline and online feature computation.

Methods

OpenMLDB tackles these challenges with several key innovations:

  • Unified Query Plan Generator: A single query plan generator supports extended SQL syntax and compilation-level optimizations (like cycle binding and compilation caching) to ensure consistent computation results across offline and online stages, reducing deployment overhead.
  • Online Real-Time Execution Engine: This engine is optimized for performance bottlenecks in ML feature computations. It uses:
  • Pre-aggregation: For long window computations, intermediate aggregation results are precomputed and cached, allowing for faster lookups during online requests.
  • Self-Adjusting Window Union: This technique dynamically adjusts task allocation and uses incremental computations to efficiently handle multi-table window unions, even with out-of-order data.
  • Offline Batch Execution Engine: This engine is optimized for high-throughput offline computations. It features:
  • Multi-Window Parallel Optimization: Allows parallel execution of independent window operations on the same table.
  • Time-Aware Data Skew Resolving: Dynamically re-partitions data based on workload and distribution to mitigate data skew issues in distributed computations.
  • Compact Time-Series Data Management: OpenMLDB employs a compact in-memory data format and stream-focused indexing to maximize memory usage and accelerate data access.

Datasets

The paper evaluates OpenMLDB using:

  • Micro-benchmark (MicroBench): A custom tool to measure feature computation performance.
  • TalkingData dataset: A publicly available dataset for evaluating memory savings.
  • Akulaku's Item Ranking service (RTP) and Geographical Location Querying service (GLQ): Real-world production use cases.
  • FEBench: A benchmark for real-time relational data feature extraction (mentioned in references).

Key Insights and Ideas

  • Unified System is Crucial: The core insight is that a single system for both offline and online feature computation drastically reduces inconsistencies and deployment friction, saving significant engineering effort.
  • Specialized Optimizations for ML Features: Standard data processing systems are not built for the unique demands of ML features (e.g., complex window functions, multi-table unions). OpenMLDB's specialized optimizations are key to achieving real-time performance.
  • Pre-aggregation for Long Windows: Pre-computing parts of long window computations is a highly effective strategy to avoid re-processing massive amounts of data for every online request.
  • Dynamic Load Balancing for Window Unions: Static partitioning for window unions is a bottleneck; dynamic adjustments are essential for handling varying workloads and data distributions.
  • Compact Data Format Matters: Significant memory savings (over 50% compared to Spark and Trino+Redis) are achieved through a specialized in-memory data format, enabling more data to be processed in memory and reducing hardware costs.

Key Results

  • Online Performance: OpenMLDB achieves 10x–20x higher online performance than Flink and DuckDB in micro-benchmarks. It also outperforms MySQL (in-memory) by over 68.4% in latency and achieves over 17x higher throughput.
  • Offline Performance: OpenMLDB delivers 6x faster offline performance than Spark and MPP databases like GreenPlum.
  • Memory Efficiency: OpenMLDB uses 40% less memory than in-memory databases like Redis and achieves over 50% memory saving compared to Trino+Redis.
  • Reduced Latency: For real-time requests, OpenMLDB achieves latencies in the milliseconds range, crucial for online ML applications. For example, a 45x reduction in response time was observed for a specific long-window query.
  • Scalability: The system demonstrates good scalability with respect to the number of threads, windows, and feature dimensions, maintaining low latency even with complex pipelines.

Limitations and Future Work

  • Limitations: While not explicitly detailed as limitations, the paper focuses heavily on performance and consistency. The complexity of managing and configuring such a specialized system might be a practical consideration for users. The paper mentions that pre-aggregation introduces a small overhead during data loading.
  • Future Work: The paper doesn't explicitly outline future work, but the ongoing development and community contributions (over 150 contributors, 1.6k stars on GitHub) suggest continuous improvement and expansion of features and optimizations. The system's architecture is designed to be extensible, hinting at future support for more complex ML functions and data types.

Summary updated

Abstract

Efficient and consistent feature computation is crucial for a wide range of online ML applications. Typically, feature computation is divided into two distinct phases, i.e., offline stage for model training and online stage for model serving. These phases often rely on execution engines with different interface languages and function implementations, causing significant inconsistencies. Moreover, many online ML features involve complex time-series computations (e.g., functions over varied-length table windows) that differ from standard streaming and analytical queries. Existing data processing systems (e.g., Spark, Flink, DuckDB) often incur multi-second latencies for these computations, making them unsuitable for real-time online ML applications that demand timely feature updates. This paper presents OpenMLDB, a feature computation system deployed in 4Paradigm's SageOne platform and over 100 real scenarios. Technically, OpenMLDB first employs a unified query plan generator for consistent computation results across the offline and online stages, significantly reducing feature deployment overhead. Second, OpenMLDB provides an online execution engine that resolves performance bottlenecks caused by long window computations (via pre-aggregation) and multi-table window unions (via data self-adjusting). It also provides a high-performance offline execution engine with window parallel optimization and time-aware data skew resolving. Third, OpenMLDB features a compact data format and stream-focused indexing to maximize memory usage and accelerate data access. Evaluations in testing and real workloads reveal significant performance improvements and resource savings compared to the baseline systems. The open community of OpenMLDB now has over 150 contributors and gained 1.6k stars on GitHub.
BibTeX
@inproceedings{zhou2025openmldb,
  author = {Xuanhe Zhou and Wei Zhou and Liguo Qi and Hao Zhang and Dihao Chen and Bingsheng He and Mian Lu and Guoliang Li and Fan Wu and Yuqiang Chen},
  title = {{OpenMLDB: A Real-Time Relational Data Feature Computation System for Online ML}},
  booktitle = {SIGMOD Conference Companion},
  year = {2025},
  doi = {10.1145/3722212.3724446},
}
Figure