Amortized Nonlinear Variable Selection with Prior-Data Fitted Networks

Jin Zhu, Junxian ZhuarXiv 2026 · arXiv

TL;DR

MambaPFN is a novel, pretrained Transformer that performs nonlinear variable selection instantly on new datasets without any task-specific retraining. By treating variable selection as a functional mapping problem and training on hundreds of thousands of synthetic datasets, it achieves state-of-the-art accuracy and a 10x speedup over existing model-agnostic methods.

What Problem Does This Paper Address?

In data science, variable selection is the process of identifying the most relevant features (the "Markov blanket") to explain a response variable while discarding noise. Traditional methods like Lasso are fast but struggle with complex, nonlinear relationships. Conversely, modern machine-learning-based selectors can handle nonlinearity but are "static"—they require a time-consuming optimization process for every single new dataset.

The authors argue that variable selection has yet to embrace the "Bitter Lesson" of AI: general methods that leverage massive computation and data tend to outperform hand-crafted, domain-specific ones. They aim to "amortize" variable selection, creating a foundation model that can look at a brand-new table of data and immediately point out the important variables.

Methods

The authors introduce MambaPFN (Model-agnostic Markov BlAnket selection with Prior-data Fitted Networks). Its core innovation lies in three areas:

  1. Amortized Learning Perspective: Instead of training a model on a dataset, they train a Transformer to be a mapping function $T: \text{Dataset} \to \text{Selection Vector}$. Once trained, this function is applied "zero-shot" to new data.
  2. Synthetic Data Engine (Algorithm 1): To train such a model, you need millions of examples where the "ground truth" important variables are known. The authors developed a graph-based scheme using Directed Acyclic Graphs (DAGs) to generate "faithful" synthetic datasets that satisfy rigorous statistical assumptions (like the Markov blanket property).
  3. Tailored Transformer Architecture: Standard Transformers aren't built for tabular data. MambaPFN uses:
  4. Entry-wise Embedding: To distinguish the response variable from covariates.
  5. Bidirectional Self-Attention: A "row-axis" attention to learn from samples and a "column-axis" attention to learn relationships between different variables.
  6. Permutation Invariance/Equivariance: Ensuring the results don't change if you shuffle the order of your rows or columns.

Datasets

  • Pretraining: A massive pool of 320,000 synthetic datasets generated via their DAG-based algorithm, featuring various nonlinear functional forms (interactions, additive models, etc.).
  • Evaluation (Synthetic): Held-out synthetic datasets where the number of observations ($n$) and variables ($p$) were varied to test scaling and robustness.
  • Evaluation (Real-world): The Facebook Comment Volume dataset (UCI repository), which contains ~41,000 observations and 53 predictors. This served as an "out-of-distribution" test, as the model was never trained on real-world social media data.

Key Insights and Ideas

  • The Power of PIPs: The model is trained to estimate Posterior Inclusion Probabilities (PIPs)—the probability that a variable belongs to the true model. This allows users to separate the "estimation" of importance from the "decision" of which threshold to use for selection.
  • Scaling Laws for Statistics: Much like Large Language Models (LLMs), MambaPFN exhibits a clear scaling law. Its performance improves predictably as you increase the number of pretraining datasets and the number of model parameters.
  • Model-Agnosticism: Because it is trained on a vast variety of functional relationships, MambaPFN doesn't assume the data is linear. It "discovers" the underlying structure of the data through the lens of the Transformer's attention mechanism.

Key Results

  • Superior Accuracy: In simulations, MambaPFN consistently achieved higher F1-scores and AUCs than established baselines like Lasso, SCAD, and even specialized nonlinear methods like STG and DFS.
  • Massive Speedup: MambaPFN is roughly 10 times faster than other model-agnostic methods (like Knockoffs or HSIC-Lasso) and remains competitive with simple linear models.
  • Real-World Parsimony: On the Facebook dataset, MambaPFN selected only 3 variables but achieved the highest predictive $R^2$, outperforming the "all features" model and other selectors by approximately 20%. It identified a "V-shaped" nonlinear relationship in comment-count changes that linear models missed.

Limitations and Future Work

  • Two-Step Pipeline: Currently, MambaPFN only selects variables; a separate model (like an MLP) must then be trained for actual prediction. The authors suggest extending the architecture to perform selection and prediction simultaneously.
  • Pretraining Overhead: While inference is nearly instantaneous, the initial pretraining of the Transformer is computationally expensive.
  • Architecture Evolution: The authors mention that while they used a Transformer, newer architectures like Selective State Space Models (Mamba) or Grouped-Query Attention could potentially make the training and inference even more efficient.

Summary updated

Abstract

Variable selection is an important task for interpreting potentially complex systems in fields such as biology, business, and sociology. The importance of variable selection has attracted considerable attention in statistics and machine learning. In this paper, we are mainly interested in model-agnostic variable selection for identifying variables that have general nonlinear dependence on the response. We propose MambaPFN, short for Model-Agnostic Markov BlAnket selection with Prior-data Fitted Networks, which is built upon the Transformer, a building block of many popular large language models. Specifically, we consider a functional-mapping perspective on variable-selection learning, in which the Transformer is treated as a learnable mapping from a dataset to its relevant variables. To learn this mapping, MambaPFN introduces three new components: (i) a faithful, rich, and scalable scheme for generating pretraining data, (ii) an objective that targets each variable's posterior inclusion probability, and (iii) a modified Transformer architecture tailored to variable selection. Once pretrained, MambaPFN is applied to new datasets without parameter updates. Empirically, MambaPFN achieves desirable variable-selection performance in simulation studies and exhibits a scaling law---more pretraining datasets and larger models can continuously improve its performance. Applied to a Facebook comments dataset, MambaPFN selects the fewest variables and achieves the highest computational efficiency and predictive performance, improving upon the best baselines by about 20%.
BibTeX
@misc{zhu2026amortized,
  author = {Jin Zhu and Junxian Zhu},
  title = {{Amortized Nonlinear Variable Selection with Prior-Data Fitted Networks}},
  year = {2026},
  eprint = {2611.04848},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04848},
}
Figure