TL;DR
DBCC introduces a database-side context compression middleware that transforms massive, redundant enterprise schemas and documentation into compact, high-density representations. By shifting from query-aware filtering to query-agnostic re-encoding, it slashes input tokens by up to two orders of magnitude and unlocks dramatic accuracy gains for Text-to-SQL systems on real-world large databases.
What Problem Does This Paper Address?
While large language models (LLMs) excel at Text-to-SQL tasks on standard academic datasets, their performance plummets when deployed on real-world enterprise databases. The authors argue that the primary bottleneck is no longer the models' reasoning ability, but rather how the database is presented to the model.
Enterprise databases are plagued by engineering noise: wide tables with repeated audit columns, massive families of sharded tables, opaque machine-generated identifiers, and verbose data dictionaries. Standard approaches rely on query-aware "schema linking" or retrieval methods to filter this raw context. However, these methods still operate on top of a representation that is structurally redundant and semantically bloated, often leading to context overflow or the "Lost-in-the-Middle" effect where LLMs miss crucial details.
Methods
The authors propose DBCC (Database Context Compression), a plug-and-play middleware grounded in a unified mathematical framework called SGCF (Support-Gain Component Factorization). SGCF optimizes a coverage objective: it replaces repeated subsets of information units across the database with a single component definition and compact references.
DBCC splits this compression into a two-phase architecture:
Phase I: Offline, Query-Agnostic Compression
This phase runs once per database and caches a highly compressed view:
- Structural Compression: Uses column-group factorization to extract identical column groups (like audit or status blocks) across wide tables. It also builds template hierarchies to compress sharded or partitioned tables using object-oriented style inheritance, storing only the structural "deltas."
- Semantic Compression: Employs an LLM to hierarchically extract concise keyword tags from verbose column descriptions and normalizes them, removing cross-column semantic repetition.
Phase II: Online, Query-Aware Purification
When a user query arrives, this lightweight phase runs:
- Evidence Purification: Rather than using standard text retrieval on external documentation, DBCC extracts precise, schema-anchored "typed atoms" (such as concept mappings, metric definitions, and value constraints) directly relevant to the user's question.
Datasets
The approach was evaluated on two major enterprise-scale Text-to-SQL benchmarks:
- Spider 2.0-Snow: A real-workflow benchmark on Snowflake containing 152 enterprise databases. The authors bucketed these databases by size: SMALL (<1,000 columns), MEDIUM (1,000–10,000 columns), and LARGE ($\ge$10,000 columns) to evaluate scaling behavior.
- BIRD: A large-scale database benchmark containing 95 databases and 1,534 development questions, utilized to evaluate cross-benchmark generalization.
Key Insights and Ideas
- The Paradigm Shift: Instead of repeatedly filtering the raw database context for every single query, systems should rewrite the database representation once on the database side—akin to building a database index or a materialized view.
- Lossless Compression via References: DBCC's compression is entirely lossless with respect to schema content. Because it relies on structured references and inheritance, any downstream system can deterministically expand the schema back to its original form if needed.
- Layer Orthogonality: Structural, semantic, and external-knowledge redundancies exist on entirely disjoint vocabularies. As a result, the compression gains from each operator stack additively rather than cannibalizing one another.
Key Results
- Massive Token Reductions: On the most challenging LARGE bucket of Spider 2.0-Snow, DBCC reduced the input context from 2.6 million tokens to just 34.7K tokens—a massive 98.7% reduction.
- Unlocking Unmanageable Databases: For LARGE databases where raw schemas completely broke LLM context windows (resulting in 0% strict recall), DBCC achieved a 56.5% strict recall rate under DeepSeek-V3.2 and 63.1% under Claude-Opus-4.7.
- Plug-and-Play Gains: When stacked on top of three recent state-of-the-art agentic Text-to-SQL systems (AutoLink, ReFoRCE, and Apex-SQL), DBCC consistently delivered a 1.8% to 1.9% absolute improvement in end-to-end Execution Accuracy (EX) while cutting token costs by up to 73.5%.
- Economic Amortization: The heavy LLM cost of offline semantic extraction pays for itself rapidly. The break-even point ($N^*$) is just 1 query for medium and large databases.
Limitations and Future Work
The authors categorized the remaining failure modes into four categories, primarily driven by:
- Discriminative Column Absorption: Occurrences where a unique column critical to a specific query gets folded into a frequent, generalized shared component, reducing its saliency to the linker.
- Inconsistent Keyword Normalization: Subtle synonym variances during the LLM keyword extraction phase that can disrupt cross-table schema linking.
To address these, the authors point toward developing query-aware extensions for the structural/semantic layers and expanding the SGCF principle to cover other database context layers like constraints, lineage, and historical query logs.
Summary updated