A Theoretical Framework for Understanding and Building "Any-task inference" Capable Foundation Models

Matthew McDermott, Florent Pollet, Payal Chandak, Gregory Kondas, Liat Friedman Antwarg, Noa Dagan, Isaac KohanearXiv 2026 · arXiv

TL;DR

This paper introduces a formal mathematical framework for "Any-Task Inference" (ATI) foundation models—models capable of performing arbitrary tasks without additional fine-tuning. By decomposing these models into a "universal task reduction" (Encoder, Model, Decoder), the authors provide a rigorous way to analyze sources of error and offer two distinct "recipes" for building these systems across diverse domains like healthcare and NLP.

What Problem Does This Paper Address?

The term "foundation model" is ubiquitous but remains theoretically fuzzy. Most definitions focus on how a model is trained (e.g., self-supervision at scale). However, the most impactful property of these models is often their ability to perform "zero-shot" or "any-task" inference—solving problems they weren't explicitly trained for without further weight updates.

Until now, there hasn't been a unified way to compare a Large Language Model (LLM) that uses prompting to a medical model that forecasts patient trajectories to a tabular model like TabPFN. This paper fills that gap by formalizing what it means to be an ATI foundation model and providing a language to describe why one might fail where another succeeds.

Methods

The authors propose a "Universal Task Reduction" lens. They argue that every ATI model can be decomposed into four functional components:

  1. Task Representation ($\rho$): How a latent, abstract task $t$ is converted into a tractable format (like a text prompt or a structured program).
  2. Encoder ($E$): Maps the input data $x$ and the task representation into a "universal task" input space.
  3. Model ($f$): Solves a "universal task" (e.g., predicting the next token or a future data sequence).
  4. Decoder ($D$): Translates the universal task output back into a specific label for the downstream task.
Figure 1: A graphical model of data samples x, tasks t, labels y, and representations of tasks induced by ρ, r.

Using this decomposition, the authors derive a comprehensive Error Decomposition Theorem. They break down the total error of a foundation model into:

  • Representation Error: Loss of information because the task description (prompt/query) is ambiguous.
  • Design Error: Fundamental limitations of the chosen "universal task" or decoding strategy.
  • Model Excess: The gap between the learned model and the theoretical ideal universal task solver.
  • Geometric Bregman Interaction: A term capturing complex interactions between these components.

Datasets

As a theoretical paper, it does not introduce new datasets. Instead, it validates the framework by mapping five existing, diverse foundation model paradigms onto its definitions:

  • Clinical LLMs: (e.g., GatorTron) which use natural language prompts.
  • EHR Autoregressive Models: (e.g., Renc et al.) which forecast future patient event streams.
  • Prior Fitted Networks (PFN): (e.g., TabPFN) which treat inference as a Bayesian posterior problem.
  • EveryQuery (EQ): A task-conditioned model for structured medical records.
  • Multi-task Models: Classical models trained on a fixed set of clinical outcomes.

Key Insights and Ideas

  • The "Universal Task" is the Core: Whether it's next-token prediction in GPT-4 or trajectory forecasting in medical AI, all ATI models work by reducing a specific question to a broader, "universal" prediction problem.
  • Zero-Shot is "Any-Task": The authors argue that "zero-shot" is a misnomer that focuses on training history. "Any-task inference" is a better descriptor of the capability we actually care about.
  • The Fairness Connection: One of the most surprising findings is that evaluating a foundation model across many tasks is mathematically identical to evaluating a single model for algorithmic fairness across different demographic groups. This means the "Impossibility Theorem" of fairness applies here: you generally cannot satisfy all performance consistency metrics across all tasks simultaneously.
  • Two Recipes for Success: The paper outlines two ways to build these models:
  • Generative Recipe: Model the full distribution of the data (e.g., "What happens next to this patient?") and then use a program to extract the answer.
  • Discriminative Recipe: Train the model directly on a massive, randomized variety of tasks so it learns to be "task-conditioned" from the start.

Key Results

  • Theorem 3.2 (Universal Reduction): Proves that any foundation model, no matter how complex, can be represented as an Encoder-Model-Decoder pipeline.
  • Theorem 4.11 (Error Decomposition): Provides the first formal "budget" for foundation model error, allowing researchers to pinpoint if a model is failing because the prompt is bad, the base model is weak, or the output parser is flawed.
  • Optimality Bounds: The authors show that under specific conditions (the "log-loss" setting), the total error is simply the sum of the task representation error and the model's error on the universal task.

Limitations and Future Work

  • Distribution Shift: The current framework assumes the tasks at test time come from the same "space" as those seen during design/training. Real-world "out-of-distribution" task performance remains a challenge to formalize.
  • Label Space: The framework currently assumes all tasks share a similar label space (e.g., all are classification). Extending this to mixed-type outputs (regression + classification) is a noted next step.
  • Computational Cost: The "Generative Recipe" often requires expensive Monte Carlo sampling at inference time, which the authors acknowledge as a practical hurdle compared to the "Discriminative Recipe."

Summary updated

Abstract

In this work, we provide a formal framework and mathematical definition for the subset of foundation models focused on enabling the property of "any-task inference" (ATI) (similar to zero-shot inference), rather than those designed for pre-training fine-tuning. To achieve this mathematical definition, we formalize the notion of a "space of tasks of interest," which we include in the problem domain, then provide an interface-based formal definition for ATI foundation models as those that map samples of data $x$ and tasks $t$ to predictions over the space of labels directly. While simple, this definition allows us to show that all ATI foundation models can be decomposed through a "universal task reduction" lens into encoder, model, and decoder aspects. We can leverage this decomposition to also decompose foundation model error into terms corresponding to different aspects of this reduction, stratified into a collection of "design terms," which are theoretical properties that can inform one of whether a particular proposed ATI foundation model can achieve optimality or not due to the structure of the proposed task reduction, and "realized model excess" terms which capture interactions between learned and programmatic components of foundation model systems. Lastly, we can use this perspective to offer two "prototypical" recipes for realizing ATI foundation models in diverse domains, which we term generative and discriminative variants of ATI foundation models. This exercise is grounded throughout by 5 real-world examples of foundation models in different settings, varying greatly in how these capabilities are realized, showing the applicability of this definition to real-world settings.
BibTeX
@misc{mcdermott2026theoretical,
  author = {Matthew McDermott and Florent Pollet and Payal Chandak and Gregory Kondas and Liat Friedman Antwarg and Noa Dagan and Isaac Kohane},
  title = {{A Theoretical Framework for Understanding and Building "Any-task inference" Capable Foundation Models}},
  year = {2026},
  eprint = {2610.40934},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40934},
}
Figure