When a team of language-model agents splits a task, for example one agent finding hotels, one checking the budget, and one ordering the route, there are many ways to divide the work that score equally well on the tasks used to test them. This mirrors a known result about learning representations without labels. If the only evidence is end-to-end success on independent example tasks, nothing tells us which agent is actually responsible for which requirement. The ambiguity stays hidden until a new combination of requirements arrives, or until something breaks and nobody can say which agent caused it. We propose to resolve it the way representation learning did, with cheap paired observations. We take a task and a copy that differs in exactly one constraint, run each agent on both, and check whether the change lands on a single agent. A decomposition is well-identified when each kind of edit reliably changes one agent's output and leaves the others alone.
In two sentences: success rates alone cannot distinguish a clean division of labour from a tangled one. Measuring how one-constraint edits spread across agents can, and it needs no labels.
Core Claim
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations proves that an objective depending only on the data distribution cannot prefer the "true" independent factors over infinitely many scrambled versions. Its 12,000-model study then shows that random seeds decide much of the outcome. Weakly-Supervised Disentanglement Without Compromises escapes this by observing pairs of samples whose hidden factors differ in only a few unknown coordinates. It detects which coordinates changed and ties the rest together. That paper also shows that, given pairs, a label-free quantity (reconstruction) becomes usable for model selection.
We carry both results into multi-agent task decomposition, which borrows the phrase "factor" but has a different structure. Here the "factors" are the task's constraints, and the "coordinates" are the agents. The analogue of the impossibility result is specific: two decompositions that compose to the same behaviour on the training distribution earn identical success rates, however responsibility is spread among their agents.
Our claim is that a measured quantity separates these decompositions and predicts what success rate cannot. We call it direct-effect localization: how concentrated, and how consistent across pairs, the effect of a one-constraint edit is over the agents. We predict it forecasts generalization to unseen constraint combinations, how easily failures can be traced to their root cause, and whether subagents can be reused on new tasks. We then use it to select and repair decompositions without labels.
What Exists and How This Differs
Current systems decompose each query in isolation, using different mechanisms.
- TDAG re-plans after every subtask and creates subagents on the fly. This cuts cascading failures from 34.78% to 4.35% on ItineraryBench.
- Agent Primitives lets an organizer model assemble Review, Voting, and Planning blocks that communicate through shared internal memory.
- Self-collaboration Code Generation via ChatGPT and MACM use fixed, hand-written roles.
All four judge a decomposition only by whether the final answer is right. We add a pre-deployment measurement of where each requirement is handled, and our work is complementary to TDAG's runtime repair.
The closest bridging work is in multi-agent reinforcement learning.
- D²TR separates task from role representations, using labels assigned by an LLM and no identifiability argument.
- DMAWM penalizes leakage between the internal states of identical agents in a learned simulator of the environment.
Neither touches heterogeneous language agents. Neither uses paired interventions: tasks that differ in one requirement, with the effect on each agent measured. Learning Geometric Representations of Objects via Interaction supplies the broader lesson that controlled interventions break the impossibility result. We apply that lesson to the task specification rather than to a physical body.
Demystifying Inductive Biases for (Beta-)VAE Based Architectures constrains our evaluation. It shows that apparent disentanglement on benchmarks came from factors happening to align with the largest pixel variance. When an imperceptible perturbation broke that alignment, β-VAE's mutual information gap on Shapes3D fell from 0.60 to 0.09. We expect a parallel artifact for LLM decomposers: splits may follow how prominently a constraint is worded rather than what it does. We therefore build the corresponding stress test before trusting any benchmark result.
Research Program
The measurement. A decomposition $D$ has agents $k=1,\dots,K$. For a task $x$, we form an edited task $x'$ by changing one clause, such as the budget, a city, or a required tool. The pairs are produced automatically by a rewriting model told to change exactly one requirement. Nobody labels which kind of requirement changed. This is the "sparse but unknown change" assumption of the weakly-supervised result.
For each agent we run a clamped intervention. Agent $k$ receives the edited task but the original messages from upstream agents. Holding upstream messages fixed separates a direct dependence on the edited constraint from change that merely propagated downstream. We record $\Delta_k(x,x')\in\{0,1\}$, which marks whether agent $k$'s output changed. Change is judged by structured-output equality where the benchmark allows it, and otherwise by an equivalence check calibrated on repeated runs of the same input, so that sampling noise does not count as change.
The score. We cluster the edits by an embedding of the edited clause into $C$ edit types; ground-truth constraint labels are used only in evaluation. Let $p_D(k\mid c)$ be the fraction of direct effects of edit type $c$ that fall on agent $k$. The localization score is
$$L(D)=\frac{1}{C}\sum_{c=1}^{C}\max_k\, p_D(k\mid c)\;-\;\lambda\,\frac{1}{|P|}\sum_{(x,x')\in P}\Big(\textstyle\sum_k \Delta_k(x,x')-1\Big)_+ .$$
Here $P$ is the set of pairs, $(\cdot)_+$ keeps only positive values, and $\lambda$ trades off the two terms.
- The first term rewards each edit type going consistently to one agent. This is identifiability up to relabelling agents, the analogue of recovering factors up to permutation.
- The second term penalizes single edits that directly touch several agents, which is the analogue of entanglement.
For deterministic pipelines, we will prove a short result. If every edit type has exactly one directly affected agent, the assignment of constraints to agents is unique up to permutation among all decompositions with the same end-to-end behaviour. The proof is a direct translation of the paired-sample argument. We do not claim more theory than this.
Using the score. First, selection. We sample many candidate decompositions by varying the organizer's prompt and temperature, adding TDAG-style plans, and adding hand-written roles. We pick the one with the highest $L$ on an unlabeled pair set.
Second, repair. When one edit type spreads across two agents, the organizer is told to give that constraint to one of them. When one agent absorbs several unrelated edit types, it is told to split that agent. We re-score after each change.
Cost. The cost is $K$ clamped calls per pair. We will report how many pairs are needed before $L$ stabilizes. The density problem of manifold methods in When Is Unsupervised Disentanglement Possible? warns that identifiability tools can need many samples. We expect far fewer here because the edits are discrete and targeted, but this is an empirical question.
Baselines
Unconstrained organizer. The naive reference is one decomposition produced by the organizer prompt with no selection, averaged over several draws. This reproduces the seed-lottery situation that Locatello et al. documented for unsupervised models.
Validation success. The strongest and most natural competitor selects among the same candidate pool by end-to-end success on a held-out set of in-distribution tasks. Every practitioner would use this. It receives the same number of model calls that $L$ consumes, spent on extra validation tasks rather than on pairs. Beating it means higher accuracy on the compositional test split and higher root-cause Hit@1 at equal call budget, with candidates matched on validation success.
TDAG. The strongest existing system is TDAG. It reports an ItineraryBench average fine-grained score of 49.08, against 43.02 for ReAct and 44.74 for ADAPT, and a TextCraft success rate of 73.5% against 52.0% for ADAPT. We rerun TDAG with our backbone models, since the reported numbers used GPT-4 and TDAG is known to degrade on smaller open models. We also run TDAG's runtime re-planning on top of an $L$-selected initial decomposition. Beating TDAG means the combination exceeds TDAG alone on the compositional split. Matching it in distribution while winning out of distribution is the predicted pattern.
LLM modularity judge. This baseline asks a strong model to rate each candidate decomposition for independence and non-overlap of roles, then picks the highest-rated one. It tests whether introspection substitutes for measurement.
Ablations.
- Unclamped $\Delta_k$ (measuring total rather than direct effect) tests whether the intervention design matters.
- Only the concentration term, or only the spread penalty, tests which part of the score carries the signal.
- Random pairs that differ in many clauses, which violate the sparse-change assumption, test whether the sparsity of edits is the active ingredient. The weakly-supervised theory predicts this ablation fails.
Why not just…?
Why not just select by validation accuracy on a larger, more diverse validation set? This is the real threat. If the validation set already covers constraint combinations, success rate partly measures localization indirectly. The impossibility analogy predicts that success on in-distribution tasks cannot rank decompositions that differ only in how they spread responsibility. It says nothing against a validation set that already includes the compositional shifts we test on. We settle this in two ways.
- We report the partial correlation between $L$ and held-out compositional accuracy after controlling for validation accuracy.
- We compute both curves, accuracy versus call budget for validation-based and $L$-based selection.
If validation selection catches up at modest budgets, the contribution shrinks to diagnosability and subagent reuse, and we will say so.
Why not just show the organizer the paired task in its prompt (contrastive prompting)? It is the cheapest way to use pairs, and we include it as a baseline: the organizer sees $x$ and $x'$ and is told to give the difference to one agent. Our expectation is that stating an intention to localize does not guarantee the agents' behaviour localizes. AgentTrace found GPT-4 reaches only 68.5% Hit@1 when reasoning about where failures originate, against 94.9% for structural tracing. That gap is evidence that models' self-reports about multi-agent structure are unreliable. If contrastive prompting reaches the same $L$ and the same out-of-distribution accuracy as measurement-based selection, measurement is unnecessary and we would report that.
Why not just rely on dynamic re-planning to absorb a bad decomposition at runtime? TDAG's own numbers show re-planning sharply cuts cascading failures. But re-planning reacts after a failure is visible. It cannot fix a decomposition whose agents silently share responsibility for a constraint and produce plausible but wrong outputs. That is the cascading-hallucination failure the multi-agent survey A survey on LLM-based multi-agent systems describes. The TDAG-plus-$L$ combination tests whether the two are complementary.
Why not just ask for independent subtasks in the prompt? The LLM-judge and naive baselines cover this. The variance-alignment stress test below is designed to show why it may look fine on standard benchmarks and fail when wording and function come apart.
Proposed Experiments
Benchmarks.
- ItineraryBench has explicit constraints (city, attractions, budget, timing) that make one-clause edits and ground-truth edit types natural.
- TextCraft edits the target item or available tools.
- CoDA-Bench (paper, dataset) tests whether data discovery separates from analysis. There, the top reported system (Mini-SWE-agent with GPT-5.5) reaches 61.1% execution accuracy overall and 49.6% on CoDA-HARD, with data discovery failing in about 20% of tasks.
For each benchmark we build a compositional split. Training and validation tasks use some constraint combinations, and test tasks recombine values unseen together. Backbones include one strong proprietary model and one open model, because TDAG's collapse on smaller models suggests decomposition quality matters most there.
Equifinality. The first decisive check is whether the motivating ambiguity exists at all. Among candidate decompositions within two points of each other on validation success, we measure the spread of compositional test accuracy and of $L$. If that spread is negligible, the premise is wrong and the project stops here.
Prediction. The second check is whether $L$, computed without labels, correlates with three outcomes:
- compositional test accuracy;
- root-cause Hit@1 on traces with injected faults, scored with the AgentTrace tracer so that the diagnostic method is fixed;
- the success of transplanting a subagent into a decomposition for a related task.
We also validate $L$ against a supervised version that uses true constraint labels. This is the analogue of checking an unsupervised selection rule against ground-truth disentanglement metrics.
Selection and repair. The third check pits $L$-based selection and repair against every baseline above, at matched call budgets.
Wording-versus-function stress test. The fourth check mirrors Zietlow et al. We rewrite ItineraryBench and CoDA-Bench prompts so that a binding constraint is stated tersely while a non-binding one gets lengthy, salient description. Then we measure whether unselected organizers re-split their agents along the verbose clause, and whether $L$-selected decompositions hold steady. A large drop for the naive organizer, and a small one for $L$-selected decompositions, would show that current benchmarks flatter prompt-based decomposition in the same way synthetic image benchmarks flattered variational autoencoders.
References
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Weakly-Supervised Disentanglement Without Compromises
- TDAG: A multi-agent framework based on dynamic Task Decomposition and Agent Generation
- Agent Primitives: Reuseable Latent Building Blocks for Multi-Agent Systems
- Self-Collaboration Code Generation via ChatGPT
- MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems
- Decentralized and Disentangled Task–Role Representation Learning for Generalizable Offline Multi-Agent Meta Reinforcement Learning
- Learning Disentangled Multi-Agent World Model for Decentralized Control
- Learning Geometric Representations of Objects via Interaction
- Demystifying Inductive Biases for (Beta-)VAE Based Architectures
- When Is Unsupervised Disentanglement Possible?
- AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems
- A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges
- CoDA-Bench: Can Code Agents Handle Data-Intensive Tasks?