Grading Affective Agents Against a Known Optimum

Many LLM agents now act on internal needs rather than instructions. They eat when hungry, rest when tired, and seek company when lonely. Every group that builds one tests it in a world it designed itself, and scores it with another LLM rating how "natural" the behavior looks. As a result the field cannot tell whether one architecture is better than another. It also cannot settle a direct disagreement in the literature: one paper finds that multi-step planning hurts need-driven agents, while three others say planning is essential. We propose generated text worlds whose hidden need dynamics are written down as equations, so the best possible behavior can be computed exactly. Any agent can then be scored by how far its choices fall short of that optimum, instead of by a judge's impression.

The point is to separate two questions that current evaluations merge. The first is whether an agent's internal state drives the right actions. The second is whether the result looks human. Once the first can be measured, it becomes possible to check whether LLM judges reward it at all. It also becomes possible to resolve the planning-versus-reacting dispute by turning a single dial, namely how unpredictable the needs are, rather than by comparing private simulators.

Core Claim

The task of choosing actions based on internal state can be benchmarked with ground truth. Generate worlds where needs follow an explicit drive-reduction model and solve each world for its optimal policy. Then show the LLM agent only a natural-language rendering of the state. Two claims are testable and new. First, rankings of existing affective architectures under this ground-truth score will differ substantially from rankings under GPT-4o "naturalness" judgments. Second, the conflict between reactive and planning agents is explained by how volatile the need dynamics are: planning helps when needs evolve predictably and hurts when shocks are frequent.

Grounding in Prior Work

The method for the hidden dynamics comes from A Reinforcement Learning Theory for Homeostatic Regulation. That paper defines a drive as a distance from the current internal state $H_t$ to a setpoint $H^*$ and defines reward as the reduction in that drive. It also proves that maximizing this reward is equivalent to minimizing discounted deviation from the setpoint. We adopt its drive, with $h_i$ the level of need $i$ and $n > m \geq 1$ shape constants:

$$d(H_t) = \Big(\sum_i |h^*_i - h_{i,t}|^n\Big)^{1/m}, \qquad r_t = d(H_t) - d(H_{t+1}).$$

That paper used this drive to explain animal behavior. We use it as a grader for LLM agents. We also replace the linear need decay used by Simulating Human-like Daily Activities with Desire-driven Autonomy (D2A) with a decay whose nonlinearity is controlled by a parameter, which addresses a limitation the file attributes to that system.

The motivation comes from three bodies of evidence:

The closest prior work in kind is Modularity benefits reinforcement learning agents with competing homeostatic drives, which tests competing drives in a 2D grid world with trained Q-learners. We differ in three ways:

  • Our agents act through language over meaningful activities.
  • We compute the optimum exactly, rather than comparing learners to each other.
  • The world family is parameterized so that one hypothesis about planning can be isolated.

Concordia supplies the runtime. However, we replace its LLM game master, the model that decides what happens in the world, with code-defined transitions, so that environment noise does not contaminate the score.

Research Program

The world generator. A world consists of a set of locations, a set of activities, and between 3 and 8 needs. The needs include physiological ones (hunger, energy) and abstract ones such as social connection and achievement, which follows the value modules in prior work. Each activity has three properties:

  • a duration,
  • an effect vector on the needs,
  • a probability of a stochastic shock, for example a cancelled meal or an unexpected visitor.

Needs decay with a tunable curvature parameter. A second parameter, volatility, sets how often shocks occur and how large they are.

The oracle. For worlds with five or fewer needs, each discretized to about ten levels, we solve the optimal policy exactly by value iteration. For larger worlds we use a long-horizon tree search as an approximate upper reference and report it as such.

What the agent sees. The agent receives only qualitative descriptions of its state, such as "nearly starved, feeling weak". This follows D2A's finding that LLMs reason poorly over raw numbers. The agent picks activities from a menu written in natural language.

The primary score. The primary score is normalized regret. Here $V^*$ is the oracle's expected discounted drive reduction, $V^{\pi}$ is the agent's, and $V^{\text{rand}}$ is that of a uniformly random policy, so 0 is optimal and 1 is random:

$$\text{nRegret}(\pi) = \frac{V^* - V^{\pi}}{V^* - V^{\text{rand}}}.$$

Secondary scores. These are all computable from logs:

  • Time to recover after a need is clamped to an extreme value. This is the perturbation test from the modularity paper.
  • A dithering rate, defined as the fraction of activities abandoned before their effect is realized. This targets the failure mode attributed to Maes.
  • Behavioral diversity.

Release. We would release the generator with seeds, frozen test splits, and the oracle values. The suite would be reproducible in the way the custom Concordia and Smallville setups were not.

Baselines

Naive references. The floor is the uniformly random policy that defines nRegret = 1. A fixed daily schedule written once at the start of each episode tests whether reacting to state matters at all.

The most dangerous simple competitor. This is a code-only greedy rule: always perform the activity that most reduces the currently most urgent need. It has no LLM. If the LLM architectures cannot beat it on the text worlds, their elaborate modules are not earning their cost. It gets the same activity menu and state, read from the numeric values instead of the text.

Published affective architectures. We reimplement each on the same base LLMs (an open model and a frontier API model), with equal token budgets per decision and equal prompt-tuning effort on a development split that is disjoint from the test seeds:

  • D2A,
  • the System 1 / System 2 design from Humanoid Agents,
  • the value-driven three-module agent,
  • a ReAct-style instruction agent,
  • a memory-driven Smallville agent (Generative Agents).

No paper in the file reports a ground-truth number for these systems. For D2A's win rates over ReAct, BabyAGI, and LLMob, no reported result was found on Lacuna. For Towards Humanoid's human-likeness scores, the exact values are likewise not reported in the file. The closest reported figures are of a different kind:

  • CitySim reports a GPT-4o activity human-likeness Likert score of 4.37 ± 0.18 versus 3.11 ± 0.18 for GeAn.
  • Humanoid Agents reports F1 ≥ 0.84 on need, emotion, and closeness labeling.
  • Humanoid Agents also reports a 156% increase in health-corrective activity time when the health need is zeroed.

Because no prior regret figure exists, "beating" a baseline means lower nRegret than that baseline with a 95% bootstrap interval over at least 200 test worlds that excludes zero.

Reinforcement-learning references. A Greatest-Mass Q-learning agent, which keeps one Q-learner per drive and sums their action values, is trained on the symbolic state. It serves as a learned reference. The modularity paper reports that this agent needs 1 step of exploration annealing versus 10,000 for a monolithic DQN, and we include that DQN as well.

Ablations. These isolate the ingredients under debate:

  • D2A without its activity-evaluation (imagined-effect) step.
  • D2A with and without a multi-step planner.
  • State shown numerically versus qualitatively.

Why not just…?

Why not just keep using GPT-4o or human raters on Concordia scenarios? Judges score whether a trace looks plausible, and the EmotionBench papers show GPT models systematically under-express negative affect. A judge may therefore prefer a calm agent that neglects its needs. We test this directly rather than assume it. GPT-4o and a human panel rate traces sampled across the regret range, and we report the rank correlation with nRegret. If the correlation is high, judges are a cheap proxy and that is a useful finding. If it is low or negative, judge-based leaderboards are measuring something other than whether internal state drives action.

Why not just train reinforcement-learning agents in the existing homeostatic grid world? The modularity paper already shows that this works for trained agents in a spatial grid. It does not address the systems people are actually building, which select actions through language with no training. Its comparison is also relative between learners rather than against an optimum. We keep its agent as a reference rather than compete with it.

Why not just validate against real human activity data, such as time-use surveys? This is the most serious alternative. It measures human-likeness, which our oracle does not. However, the human needs that would explain those traces are unobserved, and the strongest such data in the file (CitySim's mobility and well-being survey) is proprietary. Our benchmark makes the narrower but checkable claim of control correctness. We complement it by having human participants play the same text worlds. If people score far from the oracle, that is itself informative about what the optimum does and does not capture, and we will report it plainly.

Why not just extend EmotionBench to multi-turn stories? That is roughly what Regan et al. did. The outcome is still a self-reported rating with no consequences for action. Our worlds make affect matter only through what the agent does.

Decisive Experiments

The first experiment tests ranking reversal. We compute nRegret and judge scores for all architectures on the frozen split. Reversals in the ranking, or a weak judge–regret correlation, confirm the central concern behind the proposal.

The second experiment is the volatility sweep. We vary shock frequency from rare to frequent while holding everything else fixed, and run D2A with and without its planner. The hypothesis predicts a crossover: planning lowers regret at low volatility and raises it at high volatility. That would reconcile D2A's negative result with the planning-first designs. If there is no crossover, the dispute must come from something else, such as model or prompt differences, and the benchmark has still localized it.

The third experiment varies decay curvature. Agents that implicitly assume linear decay should degrade as curvature rises. The greedy rule gives a reference for how much of that degradation any reactive policy suffers.

The fourth experiment is the clamp test, which measures recovery time when one need is pinned to an extreme value. It asks whether LLM architectures show the isolation of the damaged drive reported for modular RL agents, or the collapse reported for monolithic ones.

The proposal fails if the code-only greedy rule matches the best LLM architecture across all settings. In that case the benchmark should be redesigned around richer social and abstract needs before release.