Experts Across Schemas: Agent-Guided Mixture-of-Relational-Experts for Cross-Database Generalization

Hiren Madhu, Sukanta Ganguly, Kiran Srinivasan, Smita Krishnaswamy, Rex YingarXiv 2026 · arXiv

TL;DR

MoREGT is a new Relational Deep Learning (RDL) architecture that achieves state-of-the-art performance on cross-database tasks while using 400x less pretraining data than previous foundation models. By combining a "value-free" Mixture-of-Experts transformer with a label-blind LLM agent for feature engineering, it proves that smarter inductive biases can beat raw data scaling.

What Problem Does This Paper Address?

Relational databases are the backbone of the web, but they are notoriously messy for machine learning. Every platform has a different schema: an online retailer tracks "customers" and "reviews," while a Q&A forum tracks "users" and "posts."

Figure 1: Two RelBench databases with no shared columns ask whether a user will stay active over the next three months. Left: online retailer rel-amazon (customers, products, reviews). Right: question-answering forum rel-stack (users, posts, comments, votes and other tables). In both, the evidence is the user’s history of timestamped rows linked by foreign keys.

Traditional Relational Deep Learning (RDL) models are usually "one-and-done"—trained for a specific schema and task. To create "Relational Foundation Models" that work across any database, researchers have recently turned to massive pretraining on thousands of databases. However, this paper asks a critical question: Can we achieve the same generalization through better architecture and targeted adaptation rather than just more data?

Methods

The authors introduce MoREGT (Mixture-of-Relational-Experts Graph Transformer), which attacks the problem from two angles:

1. The MoREGT Backbone

Unlike previous models that only tokenize individual cells, MoREGT represents both rows and cells as tokens. This allows the model to explicitly reason about entities (rows) and their attributes (cells).

  • Time-Aware Attention: It uses the "log-age" of rows to bias attention. A review from last week is mathematically weighted differently than one from three years ago.
  • Relationship-Aware Attention: It incorporates the names of foreign-key relationships (e.g., "buyer_id" vs "seller_id") directly into the attention mechanism.
  • Value-Free Expert Routing: This is the "secret sauce." Instead of a standard dense network, MoREGT uses a Mixture of Experts (MoE). Crucially, the "router" decides which expert to use based on metadata (the column name, data type, and age) rather than the actual data values. This allows the model to learn that "prices" in one database should be processed similarly to "costs" in another, even if it has never seen that specific schema before.
Figure 2: Overview of MoREGT. (a) Temporal neighborhood of a seed 𝑒∗at time 𝑡∗. (b) Cell and row tokens. (c) Two-level relational block, repeated 𝐿times. (d) Stage 1 pretrains on the other databases; stage 2 fine-tunes on the target database with agent-derived columns (green). (e) Expert routing reads what a cell is, never its value. (f) Attention depends on log-age and on the name of each FK role.

2. The Label-Blind LLM Agent

To handle database-specific quirks (like specific status codes or calculated rates), the authors use an LLM agent (Claude 3 Opus). The agent reads the schema and column statistics to propose new "derived columns" (e.g., pace = milliseconds / laps).

  • Label-Blind: The agent never sees the target labels or model scores, preventing data leakage.
  • Validated: Proposals are run through a "sandbox" to ensure they parse correctly and aren't redundant before being added to the model's input.
Figure 3: The agentic loop for proposing and validating derived columns.

Datasets

The model was evaluated on RelBench, a standard benchmark for relational deep learning.

  • Scope: 7 diverse databases (Amazon, StackOverflow, F1 racing, etc.) covering 21 tasks (12 classification, 9 regression).
  • Protocol: "Leave-one-database-out" (LODO) pretraining. The model is pretrained on six databases and then tested on a seventh, completely unseen schema.

Key Insights and Ideas

  • Metadata is a Strong Signal: Routing experts based on what a column represents (metadata) rather than what it contains (values) is highly effective for zero-shot transfer. It allows the model to "borrow" expertise from similar columns in other databases.
  • Inductive Bias > Scale: By explicitly modeling time and relationship roles, MoREGT outperforms models like RT-J, which was trained on roughly 400 times more databases.
  • LLMs as Feature Engineers: While RDL aims to eliminate manual feature engineering, "label-blind" agentic engineering provides a middle ground. It captures domain-specific logic that is hard for a neural network to "discover" from raw numbers alone (like specific error code ranges).

Key Results

  • State-of-the-Art: MoREGT (with the agent) achieved the highest mean scores on the RelBench leaderboard: 79.63 AUROC for classification and 26.98 nMAE for regression.
  • Efficiency: It beat the previous SOTA (RT-J) despite RT-J being trained on a corpus of ~2,400 databases compared to MoREGT's 6.
  • Ablation: Removing the time-aware attention caused the biggest performance drop, highlighting that temporal context is the most critical shared structure across relational schemas.

Limitations and Future Work

  • Agent Scope: The LLM agent is currently restricted to "single-row" formulas. It cannot yet propose complex aggregations across multiple tables (e.g., "average spend per user over the last month"), leaving that work to the transformer backbone.
  • Inference Cost: While the agent is only run once per task, the MoE architecture adds some complexity, though the authors show MoREGT is still significantly faster than "in-context" learning models like RT-J.
  • Future Direction: The authors suggest that combining these architectural improvements with the massive datasets used by other foundation models could lead to even more powerful relational representations.

Summary updated

Abstract

Web platforms record user and item activity in relational databases whose schemas differ from platform to platform, which makes it difficult to transfer predictive models between them. Relational foundation models address this through cross-database pretraining, yet recent improvements have relied on increasingly large pretraining corpora. We ask whether a better architecture, together with features specific to the target database, can deliver comparable gains from far less pretraining data. We introduce MoREGT, a mixture-of-relational-experts graph transformer that represents both rows and cells as tokens, adds the log-ages of rows and the names of foreign-key relationships to attention, and routes cells and rows to experts by column metadata rather than by values. A label-blind LLM agent complements the model: it reads the target database's schema and column statistics and adds validated single-row derived columns, without access to labels or model scores. On 21 RelBench tasks with leave-one-database-out pretraining on six databases, the MoREGT backbone improves on the Relational Transformer pretrained on the same databases by 2.25 points of mean AUROC and 5.83 points of mean nMAE. The agent's columns raise mean AUROC from 78.45 to 79.63 and lower mean nMAE from 27.91 to 26.98, the highest means on both metrics among the compared methods, including RT-J, which pretrains on about 400 times as many databases.
BibTeX
@misc{madhu2026experts,
  author = {Hiren Madhu and Sukanta Ganguly and Kiran Srinivasan and Smita Krishnaswamy and Rex Ying},
  title = {{Experts Across Schemas: Agent-Guided Mixture-of-Relational-Experts for Cross-Database Generalization}},
  year = {2026},
  eprint = {2611.18315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18315},
}
Figure