MAST-VLA: Skill-Conditioned Adaptation with Trajectory-Affordance Priors for Multi-Skill Robot Manipulation

Jiachi Yang, Zhiyi Xu, Hongquan Zhang, Xin Tan, Zhizhong Zhang, Jingyu Gong, Yuan XiearXiv 2026 · arXiv

TL;DR

MAST-VLA is a new dual-conditioning framework that significantly improves how Vision-Language-Action (VLA) models handle complex, multi-step robot manipulation. By explicitly modeling "what" skill to perform (via skill-conditioned MoE adapters) and "where" to interact (via trajectory-derived affordance priors), it achieves state-of-the-art results on the CALVIN benchmark and demonstrates robust performance on real-world hardware.

What Problem Does This Paper Address?

Generalist robot policies need to translate high-level language instructions into precise physical actions across diverse scenes. However, current VLA models often struggle with two critical forms of "grounding":

  1. Primitive-Skill Semantics: Identifying exactly which type of motion is required (e.g., "grasping" vs. "pushing").
  2. Object-Interaction Semantics: Pinpointing the exact contact regions and motion paths needed for execution.

Standard models often leave these implicit, relying on low-dimensional action supervision to figure them out. This leads to "compounding errors" in long-horizon tasks—where a small mistake in one subtask makes the next one impossible. MAST-VLA tackles this by making both the skill type and the spatial interaction area explicit components of the model's reasoning process.

Methods

The authors introduce two complementary conditioning paths into a VLA backbone (specifically, a Qwen2.5-0.5B model):

1. Skill-Conditioned LoRA-MoE

Instead of using a single set of weights for all tasks, MAST-VLA uses LoRA Mixture-of-Experts (LoRA-MoE).

  • The "What": Instructions are mapped to a primitive skill taxonomy (e.g., Rotation, Picking, Stacking).
  • The Mechanism: This skill ID retrieves specific "skill-query tokens" and "routing biases." These biases nudge the model to select the most relevant low-rank experts for that specific skill, allowing for specialized adaptation without losing general knowledge.

2. Trajectory-Affordance Spatial Prior

To help the model focus on the right parts of the image, the authors use Affordance Heatmaps.

  • The "Where": During training, the model looks at the future 3D trajectory of the robot's gripper and creates a heatmap of the scene surfaces it will get close to.
  • The Mechanism: A dedicated "Heatmap Head" learns to predict these areas from visual input. This predicted heatmap is then injected directly into the "Action Head" as a spatial attention bias, essentially telling the model: "Pay extra attention to these pixels when deciding how to move."
Overview of the MAST-VLA architecture, showing the dual paths of skill-conditioned MoE and trajectory-affordance priors.

Datasets

  • CALVIN ABC→D: A standard long-horizon benchmark. The model is trained on scenes A, B, and C, and then tested on the unseen scene D to evaluate cross-scene generalization.
  • Real-Robot Data: Collected on a Franka Research 3 platform. The authors used 80–100 demonstrations per primitive task to fine-tune the model for physical deployment.

Key Insights and Ideas

  • Soft Specialization is Key: Rather than hard-coding experts to skills, the routing bias provides a "soft" nudge. This allows the model to remain context-adaptive while still benefiting from skill-specific parameters.
  • Geometry Matters: By converting 3D trajectories into 2D image-space heatmaps, the model gains a dense geometric signal that simple (x, y, z) action coordinates can't provide.
  • Synergy of "What" and "Where": Ablation studies showed that while each component helps, the combination is most powerful. The skill conditioning helps select the right type of movement, while the affordance prior ensures that movement is grounded in the correct location.
Visualization of predicted affordance heatmaps across different skills, showing how the model focuses on interaction regions.

Key Results

  • SOTA on CALVIN: MAST-VLA achieved an Average Length of 4.65 (out of 5 subtasks), outperforming previous state-of-the-art models like FLOWER (4.54) and VLA-Adapter-Pro (4.50).
  • Robustness in Long Horizons: The performance gap was most noticeable in the later stages of a sequence (Success@5 improved from 80.0% to 84.7%), suggesting the model is much better at recovering from or avoiding the accumulation of errors.
  • Real-World Success: In physical experiments involving tasks like nesting cups and stacking blocks, MAST-VLA reached a 98.8% Success@1 and 70.6% Success@3, significantly higher than the VLA-Adapter-Pro baseline (33.8% Success@3).
Figure 2: Representative real-robot evaluation tasks. Representative test-time primitive subtasks are shown for two tabletop scenes. The examples cover diverse interaction modes, including nesting, pick-and-place, pushing, stacking, and insertion.

Limitations and Future Work

  • No Low-Level Correction: The model doesn't explicitly correct for real-time deviations between predicted and actual end-effector positions. Future work could integrate closed-loop feedback or replanning.
  • Depth Reliability: The affordance prior relies on RGB-D data. Heavy occlusion, reflective surfaces, or transparent objects can still degrade the spatial prior.
  • Scaling the Taxonomy: The current work uses a relatively compact skill taxonomy. Expanding this to a broader, more granular set of skills is a logical next step.

Summary updated

Abstract

Generalist robot manipulation requires a single vision-language-action (VLA) policy to execute diverse skills across scenes and instructions. However, current multi-skill VLA policies often leave two aspects of an instruction insufficiently modeled: which primitive skill to execute and where/how to contact and move objects. We propose MAST-VLA (Multi-skill Adaptation with Skill-conditioned experts and Trajectory-affordance priors), a dual-conditioning framework for robot manipulation. MAST-VLA introduces skill-conditioned LoRA Mixture-of-Experts (LoRA-MoE) adapters in the LLM backbone, where skill-query tokens and routing biases guide low-rank expert specialization, and converts a predicted affordance heatmap into a patch-level trajectory-affordance spatial prior for the action head. On the CALVIN ABC$\rightarrow$D long-horizon multi-skill cross-scene generalization setting, MAST-VLA achieves state-of-the-art performance, reaching an Avg. Len of 4.65 over five-subtask evaluation sequences. Real-robot multi-skill sequence experiments further validate its deployability under real perception and control noise.
BibTeX
@misc{yang2026mast,
  author = {Jiachi Yang and Zhiyi Xu and Hongquan Zhang and Xin Tan and Zhizhong Zhang and Jingyu Gong and Yuan Xie},
  title = {{MAST-VLA: Skill-Conditioned Adaptation with Trajectory-Affordance Priors for Multi-Skill Robot Manipulation}},
  year = {2026},
  eprint = {2610.32312},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32312},
}

Loading related content...

Figure