Play by the Type Rules: Inferring Constraints for LLM Functions in Declarative Programs

Parker Glenn, Alfy Samuel, Daben LiuSubmission to ICLR 2026 · Withdrawn

TL;DR

Embedding Large Language Models (LLMs) into SQL as user-defined functions often breaks because models output conversational filler or subtle formatting mismatches (e.g., "Washington D.C." vs. "Washington DC"). This paper proposes inferring type and value constraints directly from the surrounding SQL query context and enforcing them via constrained decoding in a single pass. This eliminates expensive LLM-based post-processing, yielding up to a 53% drop in latency and enabling small 3B models to rival 8B models on complex multi-hop question answering.

What Problem Does This Paper Address?

Relational database management systems (DBMS) excel at fast, deterministic data operations, while LLMs excel at unstructured semantic reasoning. Combining them by embedding LLM functions directly into declarative query languages (like SQL) seems natural.

However, SQL systems enforce strict static type checking and exact value matching. An unconstrained LLM might output "The answer is 40" instead of the integer 40, or fail to match a table entry due to slight naming variations. Prior systems solve this through multiple LLM post-processing calls—prompting another LLM to clean up the output or verify semantic equivalency against database values. In latency-critical database engines, these extra generation passes create massive performance bottlenecks.

Methods

The authors build on BlendSQL, an intermediate query dialect that compiles to standard SQL with embedded LLM user-defined functions (UDFs) like LLMQA (aggregation/reduction) and LLMMAP (row-level mapping):

  • Context-Driven Type Inference: Rather than relying on explicit type annotations or post-hoc corrections, the system inspects the SQL abstract syntax tree (AST) to deduce what type or value a function must return:
  • Comparisons like f() > 40 imply an int.
  • Interval predicates like f() BETWEEN 60.1 AND 80.3 imply a float.
  • Column comparisons like city = f() infer a Literal[...] type containing all distinct values currently residing in the database column city.
  • Constrained Decoding via Grammars & Regex: The inferred types are converted into regular expressions or finite-state constraints enforced at generation time (using the guidance library). Invalid continuations are masked out across token tries during inference.
  • Database-First Optimization: The system eagerly evaluates relational operations (joins, filters) first using standard SQL operators, and caches prompt prefixes. This leaves only distinct, pre-filtered records for the LLM to process.
                    ┌───────────────────────────────┐
                    │  SQL Query Expression Context │
                    │   WHERE city = {{ LLMQA(...) }}│
                    └───────────────┬───────────────┘
                                    │ Inferred Type Rule
                                    ▼
       ┌─────────────────────────────────────────────────────────┐
       │   Infer Allowed Values from DB: Literal['DC', 'NYC']     │
       └────────────────────────────┬────────────────────────────┘
                                    │ Constrained Masking
                                    ▼
       ┌─────────────────────────────────────────────────────────┐
       │  Single-Pass LLM Decoding (Token-Trie Masking via Regex) │
       └────────────────────────────┬────────────────────────────┘
                                    │ Clean, Guaranteed Type
                                    ▼
       ┌─────────────────────────────────────────────────────────┐
       │    Native SQL Compilation & DBMS Execution (SQLite)     │
       └─────────────────────────────────────────────────────────┘

Datasets

  • TAG-Bench: Built on top of BIRD-SQL across 5 domains. Tables average over 53,000 rows, testing the efficiency and execution latency of operations requiring parametric knowledge or reasoning outside the database.
  • HybridQA: A benchmark consisting of multi-hop questions requiring joint reasoning over Wikipedia tables and associated unstructured text passages. The authors evaluate on the first 1,000 validation set examples in an unlinked setting requiring retrieval (RAG).

Key Insights and Ideas

  • SQL predicates are free grammars: You do not need to guess how an LLM should format its output. The surrounding SQL clause (WHERE, ORDER BY, JOIN) already specifies the required type or literal domain.
  • Constrained decoding unlocks Small Language Models (SLMs): Smaller models (e.g., Llama-3.2-3B) frequently fail to follow complex formatting prompts when left unconstrained. Constraining their output distribution enables them to serve as reliable function executors without generating invalid formats.
  • Syntactic correctness $\neq$ semantic accuracy: While applying Context-Free Grammars (CFGs) to the program parsing phase eliminated most syntax errors (like missing parentheses), it did not consistently improve overall downstream accuracy. The hardest challenge remains semantic alignment (e.g., picking the right column or table), not shallow grammar compliance.

Key Results

  • 53% Lower Latency: When evaluated on TAG-Bench using quantized Llama-3.1-8B on an RTX 5080, BlendSQL programs executed in 0.76 seconds on average compared to 1.70 seconds for the state-of-the-art framework LOTUS, while cutting token consumption from 127 to 76 tokens.
  • Accuracy Improvements: On HybridQA, adding Type Hints + Constrained Decoding consistently beat unconstrained decoding and prompt-only type hints. For a 3B model executing queries parsed by a 70B model, semantic denotation accuracy rose from 38.7% to 45.3% (+6.6 points).
  • SLMs Punching Above Their Weight: A 3B executor running a 70B-generated program achieved 45.3% denotation accuracy, rivaling an 8B model running an end-to-end RAG baseline (45.6%). Looking only at successfully executed programs (excluding runtime parsing failures), the 3B executor reached 50.3%.

Limitations and Future Work

  • Semantic Parsing Failures: Roughly 10% of generated programs (102/1000) failed at runtime due to empty retrieval contexts, hallucinated column names, or invalid SQL references—issues that constrained grammars cannot catch without database schema awareness.
  • CFG Penalties on Larger Models: For Llama-3.3-70B, using CFG-constrained generation during program parsing slightly harmed performance, potentially due to grammar implementation bugs or parser fast-forward token issues.
  • Scope of Functions: The study primarily focuses on two primitive operators (LLMQA and LLMMAP); extending these inference rules to arbitrary complex user-defined routines remains open for future exploration.

Summary updated

Abstract

Integrating LLM powered operators in declarative query languages allows for the combination of cheap and interpretable functions with powerful, generalizable language model reasoning. However, in order to benefit from the optimized execution of a database query language like SQL, generated outputs must align with the rules enforced by both type checkers and database contents. Current approaches address this challenge with orchestrations consisting of many LLM-based post-processing calls to ensure alignment between generated outputs and database values, introducing performance bottlenecks. We perform a study on the ability of various sized open-source language models to both parse and execute functions within a query language based on SQL, showing that small language models can excel as function executors over hybrid data sources. Then, we propose an efficient solution to enforce the well-typedness of LLM functions, demonstrating 7% accuracy improvement on a multi-hop question answering dataset with 53% improvement in latency over comparable solutions.
BibTeX
@misc{glenn2025play,
  author = {Parker Glenn and Alfy Samuel and Daben Liu},
  title = {{Play by the Type Rules: Inferring Constraints for LLM Functions in Declarative Programs}},
  year = {2025},
  eprint = {2509.20208},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.20208},
}

Loading related content...

Figure