Unified Storage for Real Time Transactional Analytics

This direction is merged into another direction: hybrid transactional and analytical processing. Its last version stays readable and citable here.

Version 1 of 120 papers27 conceptsfrozen 2026-10-04

Historically, database architecture has been a "house divided." As Haoze Song, Feifei Li 0001, and their colleagues explain in A survey on hybrid transactional and analytical processing, we typically separate systems into two camps: OLTP (Online Transactional Processing) for fast writes and OLAP (Online Analytical Processing) for massive reads.

This separation exists because of a fundamental physical trade-off in how data is stored on a disk or in memory. Transactional systems prefer row-oriented storage, which keeps all data for a single customer or order together, making it easy to update a single record. Analytical systems prefer column-oriented storage, which groups all values for a single attribute (like "price") together, allowing the system to scan millions of rows to calculate an average without reading irrelevant data.

The research direction of Hybrid Transactional/Analytical Processing (HTAP) seeks to collapse this divide, allowing a single system to handle both workloads simultaneously without the "clunky, hours-long" ETL (Extract-Transform-Load) processes that usually bridge them.

The Intuition of Hybrid Layouts

The core challenge of HTAP is that forcing one storage format to do the other’s job usually results in a performance collapse. If you run a massive analytical query on a row-store, the system wastes time reading entire records just to access one column. If you try to update a single record in a column-store, the system must perform many small, expensive writes across different files.

To solve this, researchers like Lei Chen and Fatma Ozcan have explored hybrid storage layouts. Instead of picking one format, these systems use specific column-based configurations that minimize the trade-offs. By organizing data in a way that supports both high-frequency updates and complex analysis, these layouts allow the database to stay "fresh"—the analytics are performed on the most recent transactional data, not a stale copy from last night.

Self-Driving and Dynamic Adaptation

Because workloads change over time—perhaps a system is write-heavy during the day and read-heavy at night—static storage layouts are often insufficient. This has led to the concept of "self-driving" databases.

Andrew Pavlo and his team at Carnegie Mellon developed the Peloton system, described in Self-Driving Database Management Systems. Peloton uses deep learning to forecast upcoming workloads and automatically transitions data between row and column layouts. This "receding-horizon" control model allows the database to prepare its storage format before the workload even arrives, effectively eliminating the need for manual tuning by human administrators.

Similarly, Hongzhi Wang and Yan Wei proposed an automated selection system that uses an XGBoost-based cost model. Their approach identifies "hot" data (frequently updated) and "cold" data (historical), applying different storage models to each. By clustering columns that are frequently accessed together, the system can dynamically adapt its internal structure to match the specific query patterns of the moment.

Moving Computation to the Data

As hardware evolves, the bottleneck for HTAP is often the movement of data between storage and the CPU. In the Front Matter for VLDB 2022, Fatma Ozcan and Juliana Freire highlight a shift toward managing hybrid processing directly within "native computational storage."

The intuition here is simple: if the storage itself can perform basic filtering or aggregation, the system doesn't have to move massive amounts of data across the bus to the processor. This reduces the "interference" between the transactional engine and the analytical engine, allowing them to share the same underlying data without slowing each other down.

By combining dynamic storage layouts, machine-learning-driven automation, and hardware-level optimizations, HTAP research aims to create a "unified" database—one that provides the speed of a transactional system with the insight of an analytical one.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unified_storage_for_real_time_transactio_d71d6cdf,
  title = {Unified Storage for Real Time Transactional Analytics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-storage-for-real-time-transactional-analytics/rdr_d71d6cdf411745ed9773e87c9b2b382c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_unified_storage_for_real_time_transactio_d71d6cdf_v1,
  title = {Unified Storage for Real Time Transactional Analytics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-storage-for-real-time-transactional-analytics/rdr_d71d6cdf411745ed9773e87c9b2b382c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}