TL;DR
While transformers are sequence-generation champions, they struggle with automated planning because they waste capacity learning to ignore arbitrary object names and text orderings. By introducing a novel contrastive learning objective and stripping out positional encodings, Symmetry-Aware Transformers (SymT) achieve remarkable training stability and successfully extrapolate from simple training tasks to much harder, unseen planning problems.
What Problem Does This Paper Address?
Automated planning—finding a sequence of actions to transition from an initial state to a goal state—is fundamentally a sequence generation problem. Naturally, you would expect transformers to excel here. However, state-of-the-art models like PlanGPT hit a wall when trying to extrapolate: they can solve problems similar in size to their training data, but fail completely on larger, more complex instances.
The authors identify problem symmetries as the root cause. In classical planning, object names (e.g., truck1 vs. truck5) are just arbitrary identifiers with no inherent semantic meaning. Similarly, the order in which you list the facts (atoms) of a state doesn't change the state itself. A pure transformer lacks the inductive bias to understand this, resulting in a combinatorial explosion of equivalent representations. The model spends too much effort trying to implicitly learn that these permutations represent the exact same problem, leading to poor sample efficiency and generalization failures.
Methods
To inject symmetry awareness into transformers without sacrificing their flexibility, the authors propose Symmetry-Aware Transformers (SymT). They evaluate two flavors: SymTE (an encoder-only model for predicting heuristics) and SymTED (an encoder-decoder model for generating actual plans).
Their approach relies on two main pillars:
1. Architectural Changes for Permutation Equivariance
- Atom-Level Embeddings: Instead of tokenizing an atom into separate words (e.g.,
at,truck5,loc3), SymT maps each grounded atom into a single compositional embedding vector. - Ditching Positional Encodings: The encoder completely omits positional encodings. Because transformer layers are inherently permutation-equivariant, removing positional data guarantees that the encoder's output is entirely invariant to the arbitrary order of input atoms. For the decoder, they use NoPE (No Positional Encoding), which has been shown to improve length generalization in sequence-to-sequence reasoning tasks.
2. Symmetry-Aware Contrastive Learning
The core contribution is a novel contrastive training objective. During training, the model is fed pairs of symmetric planning problems ($X$ and $X'$) that share the exact same structure but use randomized object names.
As shown in Figure 1, the model minimizes a combined loss function featuring two unique contrastive terms:
- Attention Loss ($L_{att}$): Forces the attention scores across all layers and heads to match between the two symmetric problem descriptions.
- Hidden State Loss ($L_{hid}$): Aligns the hidden state vectors for corresponding tokens, ensuring the model encodes structural features rather than specific object names.
Datasets
The authors generated custom datasets for four widely used planning benchmarks:
- Blocksworld: Stacking blocks into a target configuration.
- Gripper: A robot moving balls between rooms.
- Visitall: An agent navigating a grid to visit every cell.
- Logistics: Moving packages across cities using trucks and airplanes.
Note on Benchmarks: The authors discovered a significant train/test overlap in PlanGPT's original datasets (e.g., in the Visitall domain, over 30% of test problems were identical to training instances). Their new datasets explicitly separate problems into validation, interpolation (distinct but same-sized problems), and extrapolation (significantly larger problems) sets to rigorously test generalization.
Key Insights and Ideas
- Contrastive Loss as a Stabilizer: Beyond improving generalization, the contrastive objective dramatically stabilizes transformer training. Without it, encoder-only and encoder-decoder models frequently suffered from sudden loss divergence (spiking to a high plateau or becoming NaN). With the contrastive loss, only 1 out of 72 models diverged.
- Algorithmic Invariance: Forcing attention maps to align across randomized object names effectively compels the transformer to learn the underlying algorithm required to solve the domain, rather than memorizing specific object configurations.
Key Results
The quantitative results highlight a massive leap in extrapolation capability over the baseline.
As detailed in Table 2, PlanGPT scores a flat 0.00 on almost all extrapolation tasks. In contrast, SymTED (especially when paired with "regrounding"—a strategy that updates the input state after each predicted action) achieves substantial coverage on larger instances:
- Gripper Extrapolation:
SymTEDwith regrounding achieves 0.79 coverage, whereas PlanGPT fails entirely. - Visitall Extrapolation:
SymTEDreaches 0.64 coverage. - Plan Quality: When SymT models successfully find a plan, the plan lengths are often optimal or near-optimal, occasionally generating shorter plans in seconds than traditional symbolic planners (like LAMA) can find in hours.
Limitations and Future Work
- The Logistics Wall: No model—including SymT—was able to extrapolate in the Logistics domain. This aligns with known theoretical expressiveness limits of Graph Neural Networks (GNNs); Logistics requires tracking complex, long-range dependencies that current architectures struggle to capture without explicit search.
- Fixed Vocabulary Size: Like PlanGPT, SymT still relies on a fixed vocabulary size, meaning it cannot handle a test problem that contains more unique objects than the vocabulary allows. Exploring representation techniques independent of token vocabularies remains an open goal for handling truly massive planning tasks.
Summary updated