This paper, presented at AAAI-26, tackles a fundamental hurdle in applying Transformer architectures to automated planning: the "symmetry" problem. While Transformers are the kings of sequence modeling, they often fail to generalize in symbolic reasoning tasks where the specific names of objects shouldn't matter.
TL;DR
Researchers from Linköping University introduced Symmetry-Aware Transformers (SymT), which use a novel contrastive learning objective to force models to ignore irrelevant details like arbitrary object names and the order of facts. This approach allows smaller models (7M–16M parameters) to significantly outperform much larger baselines (117M parameters) when solving planning problems much harder than those seen during training.
What Problem Does This Paper Address?
In automated planning, a "state" is described by a set of facts (atoms), such as at(truck1, london). To a standard Transformer, truck1 and truck2 are distinct tokens with their own learned meanings. However, in a planning problem, these are just identifiers; if you swap all instances of truck1 with truck2, the underlying logic of the problem remains identical.
This creates two major issues:
- Combinatorial Explosion: There are millions of ways to name objects and order facts that all represent the same physical situation. A "pure" Transformer tries to learn every variation individually, which is incredibly sample-inefficient.
- Extrapolation Failure: Models like PlanGPT (the current state-of-the-art baseline) often "memorize" specific object names or positions. When faced with a larger problem (e.g., more blocks or more cities) than what was in the training set, they fail completely.
Methods
The authors propose Symmetry-Aware Transformers (SymT), featuring several key innovations:
- Contrastive Learning Objective: This is the core "secret sauce." During training, the model is fed two versions of the same planning problem that differ only in object names. The loss function ($L_{att}$ and $L_{hid}$) penalizes the model if its internal attention patterns or hidden representations differ between these two versions. This forces the model to become equivariant—it learns to process the structure of the problem rather than the labels.
- Atom-Level Embeddings: Instead of tokenizing an atom like
at(pkg1, loc2)into four separate words, they embed the entire relationship into a single vector. This allows the model to handle different numbers of arguments (arity) more cleanly. - Removing Positional Encodings (NoPE): In the encoder (which processes the current state), they remove positional encodings entirely. Since the order of facts in a list doesn't change the state of the world, removing these encodings makes the model naturally "permutation-invariant."
- Architectural Variants: They tested two versions: SymT-ED (an encoder-decoder for generating full plans) and SymT-E (an encoder-only model for predicting "heuristics," or how many steps are left to reach the goal).
Datasets
The authors found that existing PlanGPT benchmarks had too much overlap between training and testing data. To truly test "extrapolation" (solving harder problems than seen before), they generated new datasets for four classic domains:
- Blocksworld: Stacking blocks in specific orders.
- Gripper: A robot moving balls between rooms.
- Visitall: An agent visiting every cell in a grid.
- Logistics: Moving packages via trucks and planes across cities.
Each dataset included an Extrapolation set containing instances significantly larger and more complex than the training data.
Key Insights and Ideas
- Inductive Bias via Loss, Not Just Architecture: While Graph Neural Networks (GNNs) have a built-in "bias" for structural data, Transformers do not. This paper shows you can "inject" that bias into a Transformer using contrastive loss, giving you the best of both worlds: the power of the Transformer and the logic of a GNN.
- Stability through Symmetry: Interestingly, the authors found that the contrastive objective made training much more stable. Models without it often suffered from "loss divergence" (crashing) during training on complex reasoning tasks.
- Heuristics are Harder than Actions: The model struggled more with predicting the "distance to goal" (heuristic) than the "next action." This is because a heuristic requires looking at the entire state, whereas a single action might only depend on a few local objects.
Key Results
- Superior Extrapolation: In the Visitall domain, SymT-ED achieved a 0.64 coverage score on extrapolation tasks, while the PlanGPT baseline scored 0.00.
- Efficiency: The SymT models are an order of magnitude smaller than PlanGPT (7M–16M parameters vs. 117M) yet solved more problems across almost all domains.
- Speed: SymT-ED can generate plans in seconds that would take classical symbolic planners (like LAMA) minutes or even hours to find for certain complex instances.
- The Logistics Wall: No model (including SymT) was able to extrapolate in the Logistics domain. This suggests there is a "complexity ceiling" for current Transformer architectures when it comes to highly nested, multi-vehicle coordination.
Limitations and Future Work
- Fixed Vocabulary: The models still rely on a fixed-size vocabulary for object names. If a problem has more objects than the vocabulary allows, the model can't process it. Future work will look at "vocabulary-independent" representation learning.
- The "Perfect" Algorithm: While SymT is much better at generalizing, it still isn't perfect. The authors suggest that combining these symmetry-aware models with traditional search algorithms (like A*) could be the key to achieving 100% reliability in automated planning.
Summary updated