TL;DR
Typed Decision Models (TDMs) offer a faster, cheaper, and often more robust alternative to Large Language Models (LLMs) for tabular data tasks like classification and anomaly detection. While TDMs are significantly less sensitive to how a question is worded, both TDMs and LLMs remain surprisingly fragile to how a table is organized—though a model's self-reported confidence is a reliable indicator of which decisions are likely to "flip" under different formatting.
What Problem Does This Paper Address?
Tabular data is the backbone of industries like finance and healthcare, yet applying modern language models to it remains tricky. Standard LLMs are designed for text generation, not categorical decision-making. When used for tables, LLMs require expensive decoding processes, and their answers often change based on trivial factors like the order of columns or the specific wording of a prompt.
This paper evaluates a new class of models called Typed Decision Models (TDMs). Unlike LLMs, TDMs return a probability distribution over a fixed set of possible answers in a single pass, avoiding the overhead of text generation. The authors investigate whether these models are practical for tabular data and, crucially, whether they are robust to "rewrites"—changes in the input format that preserve the underlying information.
Methods
The researchers conducted a head-to-head comparison between four TDMs (Jev, Clef, Kev-27B, and GPT-6 Luna Decisions) and three state-of-the-art LLMs (GPT-6 Luna, Qwen3.8-27B, and Llama-3.3-70B).
The core of their evaluation involved 21 types of "rewrites" categorized into two groups:
- Question Formulation: Renaming options (e.g., "yes/no" to "A/B"), reordering options, or paraphrasing instructions.
- Table Organization: Changing the serialization format (e.g., Markdown table vs. JSON object vs. comma-separated array) or shuffling the order of rows and columns.
They measured performance (Macro-F1, Accuracy, AUROC) and robustness, defined by the "flip rate"—how often a model changes its decision when presented with a rewritten version of the same data.
Datasets
The study utilized three diverse tabular benchmarks:
- Row Classification: 10 datasets from the TabLLM and FeatLLM benchmarks (e.g., healthcare and finance records).
- Fact Verification: The TabFact dataset, requiring models to determine if a claim about a table is supported or refuted.
- Anomaly Detection: The ReTabAD benchmark, consisting of 20 sources of normal and anomalous samples.
Key Insights and Ideas
- TDMs are "Decision-Native": Because TDMs output probabilities directly, they bypass the "text-to-option" mapping problems that plague LLMs. This makes them significantly cheaper and faster (up to 8x cheaper than LLMs in some cases).
- The Robustness Paradox: Most TDMs are remarkably robust to how you ask the question (renaming options barely affects them). However, they are just as sensitive as LLMs to how the table is organized. Simply changing a table from an array to a JSON format can flip up to 20% of a model's decisions.
- Confidence as a Shield: A major "aha moment" is that a TDM's confidence (the probability assigned to the top choice) is a strong predictor of stability. High-confidence decisions rarely flip, while low-confidence ones are highly volatile.
Key Results
- Performance: TDMs performed as well as or better than LLMs on two out of three tasks. For instance, Kev-27B led in anomaly detection, while Jev was best at fact verification.
- Cost & Latency: TDMs like Jev and GPT-6 Luna Decisions were found to be 1.6x to 8x cheaper per decision than their LLM counterparts.
- Question Robustness: Renaming or reordering options changed only 2–8% of decisions for Jev, Clef, and Kev-27B. In contrast, the same changes caused LLMs to flip up to 30% of their decisions.
- Table Fragility: No model was robust to table organization. For every model tested, at least one-third of row classification decisions changed under some rewrite of the table format or order.
- Strategic Re-querying: By identifying the 25% of rows where the model had the lowest confidence and re-querying them in different formats, the researchers recovered nearly 90% of the potential accuracy gain that would come from re-querying every single row.
Limitations and Future Work
- Table Sensitivity: The primary limitation is the lack of robustness to table organization. The authors suggest that future work should focus on training TDMs specifically on diverse table rewrites to build "format invariance."
- Fixed Choices: TDMs are currently limited to tasks with a pre-defined set of answers. They cannot provide free-text explanations or perform open-ended reasoning.
- Context Constraints: TDMs generally have shorter context windows than the largest LLMs, limiting their ability to process extremely large tables or numerous in-context examples.
- Hybrid Systems: A promising future direction is a hybrid approach: using fast TDMs for the majority of decisions and routing low-confidence cases to a more expensive LLM for detailed "review" or explanation.
Summary updated