TL;DR
While Large Language Models (LLMs) have become proficient at writing SQL queries for existing databases, they struggle significantly when tasked with designing or describing the underlying database structure (schema) itself. The authors introduce SchemaBridge, a benchmark of 902 schemata, revealing that critical structural details like data types and foreign key relationships are often lost in translation between natural language and formal code.
What Problem Does This Paper Address?
Most research into LLMs and databases focuses on Text2SQL—translating a question into a query for a database that already exists. However, in the real world, engineers first have to design the schema from a set of messy, often incomplete requirements.
The authors identify a major gap: we don't know how well LLMs can handle the bidirectional translation between natural language and Data Definition Language (DDL). Specifically, can an LLM:
- Design a schema from requirements (Text2Schema)?
- Explain an existing schema in plain English (Schema2Text)?
- Preserve the structure through a "round trip" (Schema $\rightarrow$ Text $\rightarrow$ Schema)?
The challenge lies in the different levels of abstraction. Natural language is great for business logic but terrible for specifying NOT NULL constraints or specific INT vs BIGINT types. Conversely, DDL is precise but lacks the "why" behind the design.
Methods
The researchers developed SchemaBridge, a framework designed to diagnose exactly where information is lost.
Key methodological innovations include:
- Fine-grained Fidelity Metrics: Instead of a single "accuracy" score, they measure fidelity across five dimensions: Tables, Attributes, Domain integrity (types/nullability), Entity integrity (primary keys), and Referential integrity (foreign keys).
- Anatomy of Structural Fidelity: A systematic way to count matches between a "gold" reference schema and the LLM's output.
- Masking Experiments: They hide specific parts of a schema (like table names or keys) to see if the LLM can infer them based on remaining structural cues.
- Controlled Round-Trips: By fixing either the "describer" model or the "constructor" model, they can isolate whether errors happen during verbalization or reconstruction.
Datasets
The study uses a collection of 902 schemata stratified by size and complexity:
- Enterprise-145: Large-scale, complex enterprise DDLs (e.g., e-commerce, healthcare).
- RSchema: 518 logical annotations paired with natural language requirements.
- SchemaBench: 239 project-to-schema pairs from real open-source applications.
These are categorized into a 3x3 matrix based on Size (number of tables/columns) and Constraint Complexity (foreign keys, unique constraints, etc.).
Key Insights and Ideas
- The "Naturalness" Trap: Standard NLP metrics like BLEU or ROUGE are misleading here. A "fluent" paragraph describing a database might sound great to a human but omit the specific foreign keys needed to actually build it.
- Human Requirements are Underspecified: Human-written requirements are notoriously vague. No model could achieve more than a 0.27 F1 score for referential integrity when working from original human requirements because the humans simply didn't mention the relationships clearly.
- Explicit Prompting is a Game-Changer: When LLMs are explicitly told to be "exhaustive" in their descriptions, the round-trip fidelity skyrockets. For example, GPT-5.4-mini’s referential F1 jumped from 0.228 to 0.909 just by changing the prompt to require structural detail.
- Structural Cues vs. Domain Knowledge: LLMs are surprisingly good at guessing primary keys and relationships if they can see naming patterns (like
user_idappearing in two tables), but they are very poor at remembering specific data types and nullability through a text description.
Key Results
- Top Performers: Proprietary models like Claude Sonnet 4.6 and GPT-5.4-mini generally outperformed open-weight models, particularly as schema complexity increased.
- The Complexity Ceiling: As schemas grow larger and more interconnected, fidelity drops across the board. Even the best models struggle to maintain a perfect "round trip" for enterprise-scale databases.
- Information Loss: Domain integrity (data types) is the most fragile dimension. In masking experiments, verbalization caused the retention of domain information to drop from 0.409 to a measly 0.032.
Limitations and Future Work
The authors acknowledge that their evaluation uses strict reference-based matching. In reality, two different schema designs might be functionally equivalent even if they don't match the "gold" reference exactly.
Future directions include: * Moving beyond strict matching to functional equivalence (does the generated schema support the same queries?). * Exploring agentic frameworks where LLMs can iteratively refine a schema through validation steps. * Incorporating human-in-the-loop evaluations to see if LLM-generated descriptions are actually useful for developers, not just for reconstruction tasks.
Summary updated