[Experiment, Analysis, and Benchmark] How well do LLMs comprehend and design relational database schemata?

Sriom Chakrabarti, Sean Bin Yang, Arijit Khan, Sebastian LinkarXiv 2026 · arXiv

TL;DR

While Large Language Models (LLMs) have become proficient at writing SQL queries for existing databases, they struggle significantly when tasked with designing or describing the underlying database structure (schema) itself. The authors introduce SchemaBridge, a benchmark of 902 schemata, revealing that critical structural details like data types and foreign key relationships are often lost in translation between natural language and formal code.

What Problem Does This Paper Address?

Most research into LLMs and databases focuses on Text2SQL—translating a question into a query for a database that already exists. However, in the real world, engineers first have to design the schema from a set of messy, often incomplete requirements.

The authors identify a major gap: we don't know how well LLMs can handle the bidirectional translation between natural language and Data Definition Language (DDL). Specifically, can an LLM:

  1. Design a schema from requirements (Text2Schema)?
  2. Explain an existing schema in plain English (Schema2Text)?
  3. Preserve the structure through a "round trip" (Schema $\rightarrow$ Text $\rightarrow$ Schema)?

The challenge lies in the different levels of abstraction. Natural language is great for business logic but terrible for specifying NOT NULL constraints or specific INT vs BIGINT types. Conversely, DDL is precise but lacks the "why" behind the design.

Methods

The researchers developed SchemaBridge, a framework designed to diagnose exactly where information is lost.

Overview of SchemaBridge workflow across the three core tasks: Design, Comprehension, and Preservation.

Key methodological innovations include:

  • Fine-grained Fidelity Metrics: Instead of a single "accuracy" score, they measure fidelity across five dimensions: Tables, Attributes, Domain integrity (types/nullability), Entity integrity (primary keys), and Referential integrity (foreign keys).
  • Anatomy of Structural Fidelity: A systematic way to count matches between a "gold" reference schema and the LLM's output.
  • Masking Experiments: They hide specific parts of a schema (like table names or keys) to see if the LLM can infer them based on remaining structural cues.
  • Controlled Round-Trips: By fixing either the "describer" model or the "constructor" model, they can isolate whether errors happen during verbalization or reconstruction.
Fig. 2. SchemaBridge execution and evaluation workflow. Schemata from three collections are stratified by size and constraint complexity, passed through the model zoo in the roles of describer and constructor, and scored with element-level, schema-level, and description-level metrics. Masking is applied to test structural inferability.

Datasets

The study uses a collection of 902 schemata stratified by size and complexity:

  • Enterprise-145: Large-scale, complex enterprise DDLs (e.g., e-commerce, healthcare).
  • RSchema: 518 logical annotations paired with natural language requirements.
  • SchemaBench: 239 project-to-schema pairs from real open-source applications.

These are categorized into a 3x3 matrix based on Size (number of tables/columns) and Constraint Complexity (foreign keys, unique constraints, etc.).

Key Insights and Ideas

  • The "Naturalness" Trap: Standard NLP metrics like BLEU or ROUGE are misleading here. A "fluent" paragraph describing a database might sound great to a human but omit the specific foreign keys needed to actually build it.
  • Human Requirements are Underspecified: Human-written requirements are notoriously vague. No model could achieve more than a 0.27 F1 score for referential integrity when working from original human requirements because the humans simply didn't mention the relationships clearly.
  • Explicit Prompting is a Game-Changer: When LLMs are explicitly told to be "exhaustive" in their descriptions, the round-trip fidelity skyrockets. For example, GPT-5.4-mini’s referential F1 jumped from 0.228 to 0.909 just by changing the prompt to require structural detail.
  • Structural Cues vs. Domain Knowledge: LLMs are surprisingly good at guessing primary keys and relationships if they can see naming patterns (like user_id appearing in two tables), but they are very poor at remembering specific data types and nullability through a text description.

Key Results

  • Top Performers: Proprietary models like Claude Sonnet 4.6 and GPT-5.4-mini generally outperformed open-weight models, particularly as schema complexity increased.
  • The Complexity Ceiling: As schemas grow larger and more interconnected, fidelity drops across the board. Even the best models struggle to maintain a perfect "round trip" for enterprise-scale databases.
  • Information Loss: Domain integrity (data types) is the most fragile dimension. In masking experiments, verbalization caused the retention of domain information to drop from 0.409 to a measly 0.032.

Limitations and Future Work

The authors acknowledge that their evaluation uses strict reference-based matching. In reality, two different schema designs might be functionally equivalent even if they don't match the "gold" reference exactly.

Future directions include: * Moving beyond strict matching to functional equivalence (does the generated schema support the same queries?). * Exploring agentic frameworks where LLMs can iteratively refine a schema through validation steps. * Incorporating human-in-the-loop evaluations to see if LLM-generated descriptions are actually useful for developers, not just for reconstruction tasks.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly used for database tasks, yet most work, such as Text2SQL, assumes that the relational schema already exists. We ask whether LLMs can design a schema from natural language requirements (Text2Schema), verbalize an existing schema (Schema2Text), and preserve its structure through a Schema-to-Text-to-Schema round trip. We present SchemaBridge, a benchmark of 902 schemata from three collections, stratified by size and constraint complexity, that scores fidelity separately for tables, attributes, domain (data types and nullability), entity, and referential integrity. Evaluating six open-weight and proprietary LLMs, we find that: (i) human-written requirements under-specify schemata, with no model exceeding 0.27 referential integrity F1; (ii) LLM-generated descriptions carry far more structure when explicitly asked to be complete, raising GPT-5.4-mini's referential F1 on SchemaBench from 0.228 to 0.909, even as BLEU, ROUGE-L, and BERTScore decrease; (iii) controlled experiments show that both constructor and describer choices influence round-trip fidelity under fixed settings; and (iv) masked schema experiments show that structural cues support recovery of keys and relationships, while joint data-type and nullability information is poorly retained through verbalization. Code and data are available at https://github.com/iamsriom/SchemaBridge
BibTeX
@misc{chakrabarti2026experiment,
  author = {Sriom Chakrabarti and Sean Bin Yang and Arijit Khan and Sebastian Link},
  title = {{[Experiment, Analysis, and Benchmark] How well do LLMs comprehend and design relational database schemata?}},
  year = {2026},
  eprint = {2611.17044},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17044},
}
Figure