Control when confidence is costly

Itzel Olivos-Castillo, Paul Schrater, Xaq Pitkow

Introduction

In the real world, "thinking" isn't free—it costs time, energy, and compute. Standard Artificial Intelligence models for decision-making under uncertainty, specifically Partially Observable Markov Decision Processes (POMDPs), typically assume that an agent should maintain the most accurate "belief" (posterior distribution) possible. In this conventional view, inference is a fixed subroutine, and the only costs considered are external, such as the energy spent moving or the distance from a goal.

However, real-world agents, from NASA rovers to biological brains, operate under strict computational budgets. Processing high-dimensional sensory data into a precise belief is expensive. This paper addresses the fundamental trade-off: When is it rational to be intentionally ignorant? The authors propose a framework to understand how an agent should balance task performance and movement effort against the internal cost of representing information.

The Meta-cognitive POMDP Framework

The core innovation of this work is the Meta-cognitive POMDP. Unlike standard models, it introduces a modified loss function that adds a penalty for the mutual information $I(s_t; b_t)$ between the hidden world state $s_t$ and the agent's internal belief $b_t$:

$$L = \mathbb{E} \left[ \sum_{t} (\text{State Cost} + \text{Action Cost} + \text{Inference Cost}) \right]$$

The Meta-cognitive POMDP framework (B) extends the conventional model (A) by incorporating inference costs, leading to a trade-off between information in belief and expected loss (C).

To make this mathematically tractable, the authors focus on Linear-Gaussian (LQG) environments. While standard LQG control (combining the Kalman filter and LQR) is well-understood, adding an inference cost fundamentally changes the optimization problem. Using stochastic gradient descent to optimize a strategy $\mathcal{Ж}$—which encompasses base dynamics and observation sensitivity—the authors demonstrate that when inference is "lossy" or frugal, the optimization landscape shifts from a unique solution to a structured family of equivalent strategies.

The Frugal Phase Transition

A key finding is the existence of a sharp threshold for information costs. Below this threshold, the agent behaves like a standard Bayes-optimal controller, maintaining high-fidelity beliefs. However, as inference costs increase, the agent undergoes a "phase transition" into a frugal regime. In this state, the agent strategically leaves some uncertainty unresolved to save "bits" of information.

As inference cost sensitivity increases, the system undergoes a phase transition (A) where the loss landscape shifts from a single optimal point (B) to a regime with multiple equivalent frugal strategies (C).

In the frugal regime, the optimization landscape flattens or bifurcates, revealing a "Frugal Family" of strategies. These strategies achieve identical total costs but exhibit different behaviors. For instance, some agents may be "credulous," trusting new observations implicitly and reacting sharply, while others are "skeptical," relying more on their internal models and moving more smoothly.

Key Insights: "Thinking Less, Moving More"

The transition to frugality leads to a strategy described as "thinking less, moving more." When information is expensive, agents apply stronger control gains. By physically suppressing the variance of the state through more aggressive action, the agent reduces the amount of "explaining" the inference engine must perform. Effectively, the agent trades "muscle" (action cost) for "brains" (inference cost).

Furthermore, the existence of a family of equally effective strategies provides "slack" for adaptability. An agent can select a specific strategy from the frugal family that satisfies secondary, unmodeled constraints, such as minimizing actuator wear or increasing robustness to model errors.

Results and Robustness

The framework was evaluated across several simulated environments:

  1. Scalar Linear-Gaussian Task: Used to derive analytical principles and visualize the phase transition.
  2. Meta-cognitive Cart-Pole: Frugal strategies derived from linear approximations successfully stabilized this classic nonlinear system. Despite wildly different visual trajectories, frugal agents achieved performance statistically identical to unconstrained agents within the meta-cognitive framework.
  3. Meta-cognitive Planar Drone: In a 2D drone simulation, the authors found that while "skeptical/serene" strategies were theoretically optimal, they were often brittle. A strategy that introduced slight oscillations in the controller's base dynamics proved much more robust to physical mismatches, such as changes in the drone's mass.

Conclusion and Future Work

This research moves beyond purely conceptual models of bounded rationality by providing analytical solutions for frugal inference in LQG systems. While the current work relies on linear approximations and Gaussian noise assumptions, it sets the stage for exploring truly nonlinear frugal inference. Future directions include adapting the loss function to biological metrics—such as neural spike counts—to bridge the gap between control theory and neuroscience, and investigating how agents handle non-Gaussian "outlier" noise in real-world environments.

Summary updated

BibTeX
@inproceedings{olivoscastillo2025control,
  author = {Itzel Olivos-Castillo and Paul Schrater and Xaq Pitkow},
  title = {{Control when confidence is costly}},
  booktitle = {PubMed},
  year = {2025},
}
Figure