FERN: Cross-Benchmark Table Retrieval for Text-to-SQL with Frozen Retrievers and Learned Ranking

Arjun Rao, Nishant GopinatharXiv 2026 · arXiv

In the race to build better Text-to-SQL systems, the "schema linking" or table retrieval step has become a playground for massive, hosted Large Language Models (LLMs). While effective, relying on models like Claude 3.5 to pick tables out of a catalog is expensive, slow, and often a non-starter for privacy-conscious enterprise environments.

The paper "FERN: Cross-Benchmark Table Retrieval for Text-to-SQL with Frozen Retrievers and Learned Ranking" asks a provocative question: How far can we get using "old school" feature engineering and classic machine learning instead of generative LLMs?

TL;DR

FERN (Feature-Engineered Ranking with No generative LLM) matches the performance of state-of-the-art LLM-based retrievers using frozen embedding models and a LightGBM ranker. It achieves near-SOTA recall on major benchmarks with a median latency of just 208ms and zero API costs, proving that smart feature engineering still rivals "brute force" generative AI.

What Problem Does This Paper Address?

To turn a natural language question into SQL, a system must first identify which tables in a database are relevant. In a large "pooled" catalog containing hundreds of tables across many databases, this is a needle-in-a-haystack problem.

Current top-tier systems like RASL use generative LLMs to extract keywords and select tables. However, this approach has three major drawbacks:

  1. Cost: High token usage fees.
  2. Latency: Waiting for a hosted LLM response adds significant delay.
  3. Privacy: Sending sensitive database schemas to external providers is often prohibited in enterprise settings.

FERN aims to solve these by using a local, non-generative pipeline that is fast, cheap, and private.

Methods

FERN operates in four distinct stages:

  1. Frozen Retrieval: It uses three "frozen" (non-fine-tuned) models to score every table in the catalog: BM25 (lexical search), Qwen3-Embedding, and BGE-large (dense vector search).
  2. Candidate Pooling: It fuses these scores to select the top 100 candidate tables.
  3. Feature Engineering: This is the "secret sauce." For each candidate, FERN computes up to 120 features. These include lexical overlaps, graph-based features (like PageRank and join distance), and "value" features (checking if literals in the question appear in the table's actual data).
  4. Learned Ranking: A LightGBM ranker—trained once on the BIRD dataset—reorders these 100 candidates to put the most likely tables at the very top.

The authors developed three variants (FERN-ev, FERN-q, and FERN-schema) to handle different levels of available metadata, such as whether "evidence hints" or column types are provided.

Table 3. Features. The six signals in the score and agree families are the pool score, BM25 (full and name cards), Qwen3 (full and name cards), and BGE. FERN-ev uses all 120.

Datasets

The authors evaluated FERN across a diverse set of benchmarks to test its ability to generalize:

  • BIRD: A large-scale benchmark where the ranker was trained.
  • Spider: A standard Text-to-SQL benchmark used for cross-benchmark evaluation (FERN was not trained on Spider).
  • Fiben: A financial database with "opaque" schemas (e.g., run-together uppercase table names).
  • BEAVER: A real-world enterprise benchmark involving three different data warehouses.
Table 1. Catalogs and test questions.

Key Insights and Ideas

  • The Power of the "Last Mile": While retrieval signals (like embedding scores) get you into the right neighborhood, non-retrieval features (like join-graph distance) are what actually find the "gold" tables. On BIRD, adding these features raised the complete@5 score (getting all required tables in the top 5) by 7.7 points.
  • Generalization is Real: A ranker trained only on BIRD performed exceptionally well on Spider without any tuning. This suggests that the features FERN uses capture universal patterns of how humans ask questions about relational data.
  • The "Opaque" Schema Wall: FERN struggled significantly on Fiben. Because Fiben uses unsegmented, all-caps table names (e.g., ACCOUNTSPAYABLEANDACCRUEDLIABILITIES), standard embedding models and lexical matchers fail. This highlights a specific area where LLMs' reasoning capabilities still hold a major advantage.

Key Results

  • Competitive Accuracy: On the Spider benchmark, FERN achieved a recall@5 of 96.7%, trailing the LLM-based RASL (97.0%) by only 0.3 points.
  • Speed: FERN processes a question in a median of 208.3 ms on a single A100 GPU.
  • Cost Efficiency: While RASL costs an estimated $1.85 per 100 queries, FERN’s retrieval cost is effectively zero (beyond local compute).
  • Ranking Impact: On Spider, the ranker improved the share of questions where every required table was in the top five from 83.9% to 95.6%.
Table 2. Inputs available to each system. FERN-ev, FERN-q and FERN-schema are defined in Section 4.

Limitations and Future Work

  • Candidate Ceiling: FERN can only rank what it retrieves. If the correct table isn't in the initial pool of 100 candidates, it's lost. This was a major issue on the Fiben dataset.
  • Schema Readability: The system relies heavily on readable English table and column names. Future work could involve using local LLMs to "clean" or segment opaque schemas before retrieval.
  • Scale: While tested on hundreds of tables, massive enterprise environments with thousands of tables may require more sophisticated initial candidate-pooling strategies.

Summary updated

Abstract

Text-to-SQL systems over large pooled catalogs must first retrieve the tables a question needs. We compare against the published table recall of RASL, which uses Claude 3.5 models to extract keywords and select tables, on pooled BIRD, Spider, and Fiben catalogs. FERN (Feature-Engineered Ranking with No generative LLM) instead fuses three frozen retrievers into 100 candidates, describes each with up to 120 features, and reorders them with LightGBM rankers fitted once on BIRD's training questions. On Spider, where it was not fitted, its candidate pool alone reaches RASL's published recall at fifteen tables (98.0), and the ranker raises the share of questions with every gold table in the top five (complete@5) from 83.9% to 95.6%. At five tables, FERN's recall is within one point of RASL's on Spider (96.7 vs 97.0) and on BIRD with its evidence hints (96.6 vs 97.5), and 8.2 points below on BIRD without them (89.3). On 121 real BEAVER questions, without BEAVER fitting, its recall at fifteen (82.1) falls between two BEAVER-trained retrievers (80.8, 84.5), but its complete@15 is below both (49.6 vs 54.5 and 62.0). On Fiben, it falls far short at fifteen tables (54.2 vs 69.2, different question sets), although its recall at fifty is numerically close to RASL's question-only retrieval result (80.7 vs 79.4). On BIRD dev with evidence, non-retrieval features raise complete@5 by 7.7 points over retrieval signals alone. On BIRD dev, median latency is 208 ms per question on one A100, without hosted model calls.
BibTeX
@misc{rao2026fern,
  author = {Arjun Rao and Nishant Gopinath},
  title = {{FERN: Cross-Benchmark Table Retrieval for Text-to-SQL with Frozen Retrievers and Learned Ranking}},
  year = {2026},
  eprint = {2611.14477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14477},
}
Figure