Factories that make many product variants in small batches lose most of their automation budget to reconfiguration. Every new part geometry means someone re-plans the order in which parts go together and re-tunes the motion planner that finds how each part slides into place. Automatic assembly planners exist, but each one is good at a different kind of step. Contact-guided physics search handles tight screws and press-fits. Free-space sampling handles long, open detours. Stability checkers catch sub-assemblies that would fall over. A real product mixes all three kinds of step. Today a person effectively decides which planner to trust for which product, which is the same manual reconfiguration that flexible automation is supposed to remove.
We propose to treat assembly planning the way recent multi-agent language-model systems treat hard reasoning problems. The problem is split into roles: a proposer, several interchangeable movers, a judge, and a diagnostician. A scheduler learns, step by step inside one search tree, which specialist should attempt each part removal. Failures are written into a shared memory as geometric constraints, such as "part 7 is blocked along +z until part 3 is gone," so every specialist can use them. The claim is testable in two sentences. On mixed-difficulty assemblies, per-step routing among heterogeneous planners should solve more products per unit of compute than the best single planner or a portfolio that runs whole planners independently. It should also do so without per-product tuning.
Core Claim
The multi-agent literature's strongest lessons transfer to physical assembly planning only after the language models are removed from the geometric loop. Those lessons are separating proposal from validation and drawing on a heterogeneous pool instead of resampling one model. Three findings motivate this. Beyond Lines and Circles shows that monolithic language models are "spatially blind" on multi-step geometric constructions, and that splitting work between solver and validator agents with tool access roughly doubles pass@1, from 13.6% to 28.1%. SYMPHONY shows that scheduling a heterogeneous pool of models at each node of a tree search, using an upper-confidence bandit, beats repeated sampling from one model. However, SYMPHONY relies on deterministic, discrete feedback such as unit tests. On the assembly side, the planners' failure modes are complementary. Assemble Them All (ATA), which simulates disassembly with unit forces in a signed-distance-field physics engine, reaches 70.8% on rotational assemblies but struggles on "maze-like" paths through open space. ASAP, which uses a graph neural network to guide a depth-first disassembly tree with a greedy gravity-stability check, lifts success to 82.08% where ATA gets 56.67%. Planning Assembly Sequence with Graph Transformer captures global part ordering but degrades with part count.
What is new here is the unit of decomposition. We do not pick a planner per product. We decompose each product's plan into individual part-removal subproblems and learn which specialist to send to each one. The feedback is noisy, continuous, and simulation-based, which is exactly the regime SYMPHONY leaves open. The inter-agent message is a structured geometric constraint rather than free text, which avoids the context growth and bias propagation that plague language-model dialogue.
Research Program
The system keeps a single disassembly tree. Each node is a partial assembly, and each edge removes one part. Four roles act on it.
The proposer ranks candidate parts to remove next. It reuses ASAP's graph network or the graph-transformer encoder from the LEGO planner, and it is interchangeable in ablations.
The movers try to find a removal path for the chosen part. The pool contains ATA's contact-guided physics search, a sampling planner in the style of the RRT-Connect stage of PAST's pipeline, and a cheap straight-line translation test.
The judge applies ASAP's greedy stability checker and a collision check to the remaining sub-assembly.
The diagnostician turns every failed attempt into a constraint record: which part, which directions were tried, which other parts caused contact, and whether the failure was geometric or gravitational. It writes the record to a pool-wide store that every mover and the proposer read before acting. This is SYMPHONY's shared reflection memory, rewritten as machine-checkable geometry.
The scheduler chooses a mover for each (node, part) pair using features that are cheap to compute before any attempt. These include the minimum clearance to neighboring parts from the signed distance field, a rotational-symmetry indicator, the free-space volume around the part, and the constraint records already attached to it. The scheduler scores mover $k$ as
$$ s_k = \hat{p}_k(x) - \lambda\, \hat{c}_k(x) + \beta \sqrt{\frac{\ln N}{n_k(x)}}, $$
where $x$ is the feature vector, $\hat{p}_k(x)$ is a learned estimate of mover $k$'s success probability, and $\hat{c}_k(x)$ is its expected wall-clock cost. The cost term matters because ATA-style simulation can take over 24 seconds per sequence inference where learned inference takes 52 ms, per PAST. The remaining symbols are $\lambda$, a cost weight; $N$, the total number of attempts so far; $n_k(x)$, the number of attempts by mover $k$ in the feature neighborhood of $x$; and $\beta$, an exploration constant. The success model is trained on logged attempts from the ASAP training split and then updated online within each product. That online update is what should remove per-product tuning.
The analysis side asks how often a solved plan actually uses more than one mover. It also asks which feature best predicts the right specialist, and whether structured constraints transfer across nearby product variants, so that re-planning a perturbed variant is cheaper than planning from scratch.
Baselines
The naive reference is a single geometric sampling planner with random part ordering. ATA reports that traditional geometric planners reach about 6.9% on its rotational subset, and ASAP reports 51.25% for a genetic-algorithm sequencer on its 240-assembly test set. These bound what uninformed search achieves.
The strongest existing method is ASAP's learning variant. ASAP reports an 82.08% success rate on its 240-assembly test set under its high planning budget, against 56.67% for ATA. It is the right competitor because it already combines a learned proposer, a physics mover, and a stability judge, but with one fixed mover. We will run it and our system on the same 240 test assemblies, with the same training split for any learned component, the same simulator, and matched wall-clock and simulator-call budgets. Budgets will be swept so that we compare full success-versus-compute curves rather than single points. Beating it means higher success at equal budget, or equal success at substantially lower budget, with the gap concentrated on assemblies that mix tight and open-space steps.
On the rotational subset, ATA itself is the strongest reported baseline at 70.8%.
The compute-matched portfolio runs ATA, ASAP, and the sampling planner as whole, independent solvers in parallel and takes the first success. It is the most dangerous competitor, and it isolates whether per-step mixing matters beyond per-product selection.
Three ablations isolate the key ingredients. The first replaces the learned scheduler with a hand-written dispatch rule: physics search below a clearance threshold, sampling above it. The second removes the shared constraint store, so movers do not see each other's failures. The third fixes a single mover while keeping everything else.
Why not just…?
Why not just run the planners as a portfolio and keep whichever finishes first? This is a real threat. If products tend to be homogeneous, meaning all tight or all open, then choosing a planner per product captures most of the benefit, and per-step routing adds only overhead. The experiment settles it directly. We measure the fraction of our solved plans that use two or more movers, and we compare success at matched compute against the portfolio. If mixed-mover plans are rare or the curves overlap, the core claim is false, and we would report the scheduler only as a cost-saving device.
Why not just give ASAP a bigger budget or a better graph network? ASAP's proposer decides which part to remove, not how to remove it. Its mover inherits ATA's documented weakness on maze-like paths, and more search budget does not change which mover is used. The budget sweep in the baselines will show whether ASAP's curve keeps rising toward ours or plateaus. We will also report results on a constructed maze-heavy subset, where the plateau hypothesis predicts the largest gap.
Why not just use a language-model multi-agent planner, as in Simulacra or SYMPHONY? Beyond Lines and Circles shows that even with role decomposition, language models reach only about human-student levels on 2D ruler-and-compass puzzles. 3D contact-rich removal is harder. Iterative dialogue also inflates token cost. We keep a small language-model arm in which a language model plays the diagnostician and scheduler roles, reading constraint records as text, on a 50-assembly subset. This tests whether semantic reasoning adds anything over the numeric scheduler. We expect it not to, and we will report the result either way.
Why not just hand-code the dispatch rule? A clearance threshold may capture most of the routing signal. That is why it is an explicit ablation. If it matches the learned scheduler, the contribution shrinks to the shared-constraint architecture plus a simple rule, which is still useful but a narrower claim.
Proposed Experiments
The main experiment uses the ASAP 240-assembly test set, drawn from the Fusion 360 Gallery. We report success rate against compute and wall-clock planning time; ASAP's 82.08% is the target.
The second benchmark is the 24-assembly rotational subset of Assemble Them All, where ATA reports 70.8%.
The third benchmark is a maze-heavy subset that we build from the 12,970-assembly Assemble Them All pool. It contains assemblies where ATA fails within budget but the sampling planner succeeds on at least one removal step. No reported result exists on Lacuna for this subset, so all methods are run by us.
To address automation rigidity directly, we build variant families by perturbing clearances and feature dimensions of test assemblies by small amounts. We then measure success and re-planning time on each variant, with and without constraints carried over from the parent product, under no retuning. The AutoMate observation motivates reporting results on internal stair-like features as a separate stratum: its insertion policies fail on roughly 20% of geometries with such features.
The decisive checks are three. First, the matched-compute comparison against the portfolio. Second, the fraction of solved plans that mix movers. Third, the removal of the constraint store. If the scheduler beats ASAP and the portfolio only through the cost term and not through mixing, we will say so. That outcome would reframe the work as cost-aware planner selection rather than step-level decomposition.
References
- Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models
- SYMPHONY: Synergistic Multi-agent Planning with Heterogeneous Language Model Assembly
- Assemble Them All: Physics-Based Planning for Generalizable Assembly by Disassembly
- ASAP: Automated Sequence Planning for Complex Robotic Assembly with Physical Feasibility
- Planning Assembly Sequence with Graph Transformer
- Multi-level Reasoning for Robotic Assembly: From Sequence Inference to Contact Selection
- AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries