The Case for Instance-Optimized LLMs in OLAP Databases

Bardia Mohammadi, Laurent BindschaedlerarXiv 2025 · arXiv

TL;DR

Integrating Large Language Models (LLMs) directly into database queries enables powerful data cleaning and text analysis, but running them row-by-row across millions of rows is prohibitively expensive. This paper introduces IOLM-DB, a system that generates highly compressed, query-specific LLMs on the fly using quantization, sparsification, and structural pruning. By tailoring the model to the exact query and data distribution, IOLM-DB slashes memory footprints and boosts throughput by up to 3.31× while maintaining baseline accuracy.

What Problem Does This Paper Address?

Data analysts frequently need to perform semantic operations on unstructured or semi-structured data inside databases—such as summarizing product reviews, fixing typos in code repositories, or performing fuzzy joins across mismatched text fields. While modern LLMs excel at these tasks, executing an LLM prompt for every single row in a massive OLAP (Online Analytical Processing) table introduces massive computational and memory overhead.

Using general-purpose, cloud-based LLMs is too expensive and slow at scale, while running local models requires massive GPU resources that struggle with high parallelism. The authors address this bottleneck by asking: Can we optimize and shrink the LLM specifically for the single query it needs to execute?

Methods

The authors present IOLM-DB, a proof-of-concept system built on Python’s pandas library that introduces instance-optimized LLMs for databases. Instead of relying on a static, one-size-fits-all model, IOLM-DB intercepts LLM prompts within a query and dynamically generates a lightweight, specialized model tailored to that specific workload and data distribution.

The optimization pipeline uses a small sample of unlabeled calibration data from the target dataset and combines three core compression techniques:

  1. Quantization: Lowering numerical precision to 8-bit weights and activations to reduce GPU memory usage and boost throughput.
  2. Sparsification: Utilizing algorithms like SparseGPT to introduce strategic zero elements into the model weights, minimizing computational complexity.
  3. Structural Pruning: Removing non-essential model components (like specific attention heads or entire layers) using tools like LLM-Pruner to create ultra-compact architectures.

At runtime, IOLM-DB further accelerates execution using traditional database optimizations like caching (to avoid recomputing duplicate inputs) and batching (to amortize invocation overhead across multiple rows).

Datasets

The system was evaluated on three distinct real-world workloads:

  • Text Summarization: Condensing verbose text fields using the unstructured Amazon Reviews dataset.
  • Data Correction: Fixing misspellings and grammatical errors using the GitHub Typo Corpus.
  • Fuzzy Joins: Mapping semantically similar but textually mismatched entries across multiple tables.

Key Insights and Ideas

  • OLAP as the Perfect Sandbox for Compression: Analytical database workflows are highly predictable. Because the query structure and data characteristics are known before full execution, the system can reliably extract calibration data to aggressively compress the model without risking catastrophic accuracy loss on unseen data types.
  • The Performance-Accuracy Trade-off: The authors designed two variants: IOLM-DB-Perf (optimized for maximum throughput) and IOLM-DB-Acc (optimized for maximum accuracy). This gives users the flexibility to choose between rapid, slightly less accurate results or highly precise execution.
  • Unexpected Quality Boosts: Interestingly, the authors noted that tailoring the model to a specific data distribution sometimes improved the quality of the outputs compared to the general baseline, as the specialized model focused purely on the task at hand rather than distracting general-world knowledge.

Key Results

The evaluation was conducted using Meta's Llama-3.1-Instruct-8B as the baseline model on an NVIDIA Hopper H100 GPU.

  • Model Size Reduction: IOLM-DB successfully compressed the model footprint from 14.98 GB down to 8.48 GB, representing a significant reduction in GPU memory pressure and allowing for much higher parallelism.
  • Throughput Gains:
  • Summarization: Achieved up to a 3.31× throughput improvement (15.50 rows/s vs. 4.67 rows/s for the baseline).
  • Data Correction: Delivered a 2.78× speedup (7.60 rows/s vs. 2.73 rows/s).
  • Fuzzy Joins: Provided a 2.52× speedup (37.72 rows/s vs. 14.92 rows/s).
  • Accuracy Preservation: Across the Data Correction and Fuzzy Join tasks, both the performance and accuracy variants maintained a perfect normalized accuracy score of 1.0 relative to the baseline.

The exact breakdown of these trade-offs can be seen in the authors' evaluation table:

Table 1: Performance, model size, and accuracy score across workloads.

Limitations and Future Work

While promising, the current prototype has a few hurdles to clear before mainstream adoption:

  • Generation Overhead: It currently takes "single-digit minutes" to generate the instance-optimized model on the fly. While acceptable for massive, long-running OLAP queries, this overhead is too high for smaller tables or interactive, real-time querying.
  • Cache Bottlenecks: The space consumed by the vLLM cache emerged as a key memory bottleneck, limiting potential parallelism.
  • Pandas Integration: The current system interfaces with pandas rather than a native SQL engine, meaning data transfer overheads between the database layer and the custom LLM operators still need to be reengineered for production environments.

Future work will focus on advanced knowledge distillation, better cache management, and "cascading" strategies where a coarse, cheap LLM filters data before passing complex rows to a higher-precision specialized model.

Summary updated

Abstract

Large Language Models (LLMs) can enhance analytics systems with powerful data summarization, cleaning, and semantic transformation capabilities. However, deploying LLMs at scale -- processing millions to billions of rows -- remains prohibitively expensive in computation and memory. We present IOLM-DB, a novel system that makes LLM-enhanced database queries practical through query-specific model optimization. Instead of using general-purpose LLMs, IOLM-DB generates lightweight, specialized models tailored to each query's specific needs using representative data samples. IOLM-DB reduces model footprints by up to 76% and increases throughput by up to 3.31$\times$ while maintaining accuracy through aggressive compression techniques, including quantization, sparsification, and structural pruning. We further show how our approach enables higher parallelism on existing hardware and seamlessly supports caching and batching strategies to reduce overheads. Our prototype demonstrates that leveraging LLM queries inside analytics systems is feasible at scale, opening new possibilities for future OLAP applications.
BibTeX
@misc{mohammadi2025case,
  author = {Bardia Mohammadi and Laurent Bindschaedler},
  title = {{The Case for Instance-Optimized LLMs in OLAP Databases}},
  year = {2025},
  eprint = {2507.04967},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.04967},
}

Loading related content...

Figure