Typed Decision Models on Tabular Data

Baowen Zhang, Hangting Ye, Hongyuan ZhaarXiv 2026 · arXiv

TL;DR

Typed Decision Models (TDMs) offer a faster, cheaper, and often more robust alternative to Large Language Models (LLMs) for tabular data tasks like classification and anomaly detection. While TDMs are significantly less sensitive to how a question is worded, both TDMs and LLMs remain surprisingly fragile to how a table is organized—though a model's self-reported confidence is a reliable indicator of which decisions are likely to "flip" under different formatting.

What Problem Does This Paper Address?

Tabular data is the backbone of industries like finance and healthcare, yet applying modern language models to it remains tricky. Standard LLMs are designed for text generation, not categorical decision-making. When used for tables, LLMs require expensive decoding processes, and their answers often change based on trivial factors like the order of columns or the specific wording of a prompt.

This paper evaluates a new class of models called Typed Decision Models (TDMs). Unlike LLMs, TDMs return a probability distribution over a fixed set of possible answers in a single pass, avoiding the overhead of text generation. The authors investigate whether these models are practical for tabular data and, crucially, whether they are robust to "rewrites"—changes in the input format that preserve the underlying information.

Methods

The researchers conducted a head-to-head comparison between four TDMs (Jev, Clef, Kev-27B, and GPT-6 Luna Decisions) and three state-of-the-art LLMs (GPT-6 Luna, Qwen3.8-27B, and Llama-3.3-70B).

The core of their evaluation involved 21 types of "rewrites" categorized into two groups:

  1. Question Formulation: Renaming options (e.g., "yes/no" to "A/B"), reordering options, or paraphrasing instructions.
  2. Table Organization: Changing the serialization format (e.g., Markdown table vs. JSON object vs. comma-separated array) or shuffling the order of rows and columns.

They measured performance (Macro-F1, Accuracy, AUROC) and robustness, defined by the "flip rate"—how often a model changes its decision when presented with a rewritten version of the same data.

Datasets

The study utilized three diverse tabular benchmarks:

  • Row Classification: 10 datasets from the TabLLM and FeatLLM benchmarks (e.g., healthcare and finance records).
  • Fact Verification: The TabFact dataset, requiring models to determine if a claim about a table is supported or refuted.
  • Anomaly Detection: The ReTabAD benchmark, consisting of 20 sources of normal and anomalous samples.

Key Insights and Ideas

  • TDMs are "Decision-Native": Because TDMs output probabilities directly, they bypass the "text-to-option" mapping problems that plague LLMs. This makes them significantly cheaper and faster (up to 8x cheaper than LLMs in some cases).
  • The Robustness Paradox: Most TDMs are remarkably robust to how you ask the question (renaming options barely affects them). However, they are just as sensitive as LLMs to how the table is organized. Simply changing a table from an array to a JSON format can flip up to 20% of a model's decisions.
  • Confidence as a Shield: A major "aha moment" is that a TDM's confidence (the probability assigned to the top choice) is a strong predictor of stability. High-confidence decisions rarely flip, while low-confidence ones are highly volatile.
Performance of different models across tasks as in-context examples increase.

Key Results

  • Performance: TDMs performed as well as or better than LLMs on two out of three tasks. For instance, Kev-27B led in anomaly detection, while Jev was best at fact verification.
  • Cost & Latency: TDMs like Jev and GPT-6 Luna Decisions were found to be 1.6x to 8x cheaper per decision than their LLM counterparts.
  • Question Robustness: Renaming or reordering options changed only 2–8% of decisions for Jev, Clef, and Kev-27B. In contrast, the same changes caused LLMs to flip up to 30% of their decisions.
  • Table Fragility: No model was robust to table organization. For every model tested, at least one-third of row classification decisions changed under some rewrite of the table format or order.
Comparison of decision flips (left) and Macro-F1 changes (right) across different rewrite types.
  • Strategic Re-querying: By identifying the 25% of rows where the model had the lowest confidence and re-querying them in different formats, the researchers recovered nearly 90% of the potential accuracy gain that would come from re-querying every single row.
Figure 4: Re-querying the least confident rows (row classification). For the x% of rows with the lowest confidence, the answer in the original format is replaced by the combination of the answers in the original and the four other formats. The y-axis gives the share of the Macro-F1 gain obtained when every row is re-queried, and the legend gives that gain in points.

Limitations and Future Work

  • Table Sensitivity: The primary limitation is the lack of robustness to table organization. The authors suggest that future work should focus on training TDMs specifically on diverse table rewrites to build "format invariance."
  • Fixed Choices: TDMs are currently limited to tasks with a pre-defined set of answers. They cannot provide free-text explanations or perform open-ended reasoning.
  • Context Constraints: TDMs generally have shorter context windows than the largest LLMs, limiting their ability to process extremely large tables or numerous in-context examples.
  • Hybrid Systems: A promising future direction is a hybrid approach: using fast TDMs for the majority of decisions and routing low-confidence cases to a more expensive LLM for detailed "review" or explanation.

Summary updated

Abstract

Typed decision models are language models that answer a question with a fixed set of possible answers by returning a probability for each answer instead of generating text. We study them on tabular data and compare four decision models with three large language models (LLMs) on row classification, fact verification and anomaly detection. Every request is sent in its original form and under 21 types of rewrite that keep its content, such as renaming the options or writing the table in another format. The decision models perform as well as the LLMs at a lower cost. Three of them are also far more robust to how the question is formulated. No model we test is robust to how the table is organized. For every model, at least a third of the row decisions change under some rewrite of the table or the question. The decisions to which a decision model gives low probability are the ones most likely to change. Decision models are a practical choice for tabular decisions, but their robustness remains an open problem.
BibTeX
@misc{zhang2026typed,
  author = {Baowen Zhang and Hangting Ye and Hongyuan Zha},
  title = {{Typed Decision Models on Tabular Data}},
  year = {2026},
  eprint = {2610.19536},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.19536},
}
Figure