GeneralVLA-2: Geometry-Aware Reconstruction and Governed Memory for Robot Planning

Haoyu Wang, Guoqing Ma, Zeyu Zhang, Yandong Guo, Boxin Shi, Hao TangarXiv 2026 · arXiv

TL;DR

GeneralVLA-2 introduces a powerful upgrade to hierarchical vision-language-action (VLA) robotics systems by fixing two critical bottlenecks: hallucinated 3D object geometry and unreliable experience reuse. By combining a mask-verified multi-view reconstruction branch (GeoFuse-MV3D) with a highly structured, Governed KnowledgeBank memory system, it enables safer, training-free, and more precise robot trajectory planning in both simulation and real-world environments.

What Problem Does This Paper Address?

Hierarchical VLA systems are highly attractive for robotics because they break down complex tasks into interpretable stages: perception, spatial reasoning (planning), and low-level control. However, their performance hinges heavily on the quality of the data fed to the mid-level planner. The authors identify two major pain points in existing architectures:

  1. Perception Hallucinations: Single-view or naive monocular 3D object reconstruction methods often hallucinate an object's pose or backside structure. For a robot arm, a few centimeters of error in geometry or pose estimation means the difference between a successful grasp and a catastrophic collision.
  2. Ungoverned Memory Retrieval: Previous long-term memory systems for agents rely almost entirely on simple semantic text similarity. If a robot faces a new task, it might retrieve a past experience that sounds similar but is geometrically irrelevant, or worse, blindly replay a strategy that previously failed.

Methods

GeneralVLA-2 addresses these challenges by restructuring the inputs that condition its 3D planning agent.

Figure 1: Overview of GeneralVLA-2 for robot manipulation. When calibrated multi-view ob- ject observations are available, GeoFuse-MV3D converts the views, masks, and camera poses into refined object-centric 3D evidence; the 3D-capable planning agent combines this evidence with gov- erned KnowledgeBank retrieval; and the robot execution module follows the resulting multi-stage end-effector trajectory.

1. GeoFuse-MV3D Reconstruction Branch

When a robot can capture multiple short views around an object, it shouldn't have to guess its shape. GeoFuse-MV3D refines a baseline 3D Gaussian Splatting representation using two distinct geometry sources:

  • Source A (Geometry-Prior Provider): Integrates external feed-forward 3D estimates (via VGGT) and applies an appearance calibration.
  • Source B (Provider-Free Axis Compensation): Uses only the input masks and camera poses to calculate axis-wise scale and shift corrections.

Instead of aggressively deleting low-confidence points, the system applies a soft visual-hull refinement that gently shrinks low-support points inward. Crucially, it performs geometry-only fusion, blending the spatial $(x,y,z)$ coordinates while locking the original high-fidelity appearance fields (color, opacity, and spherical harmonics) to prevent visual drift.

Figure 2: GeoFuse-MV3D reconstruction branch in GeneralVLA-2. The branch keeps the same multi-view inputs, masks, and poses as MV-SAM3D, then refines the baseline with two comple- mentary geometry sources and conservative geometry-only fusion.

2. Governed KnowledgeBank

The memory cache is upgraded into a rigorous database where every experience is stored as a structured record containing quality scores, confidence levels, lifecycle states (provisional, active, summary, archived), and conflict links.

  • LLM-as-Verifier: After a task, a verifier evaluates the trajectory based on task completion, collision safety, and execution validity.
  • Precision Retriever: Rather than just matching text, the retrieval function balances semantic relevance with success priors, recency, and explicit penalties for stale or conflicting records.

Datasets

The authors evaluated the individual modules and the unified system across several distinct benchmarks:

  • GSO-30: A 30-object multi-view dataset derived from Google Scanned Objects, used to benchmark the 3D reconstruction pipeline.
  • Terminal-Bench 2.0 & SWE-Bench Verified: Used to validate the governed memory module's performance on long-horizon software agent tasks independent of the physical robot stack.
  • RLBench: 14 diverse simulation manipulation tasks (e.g., Close Box, Open Jar, Play Jenga) used for training-free robotic planning evaluation.
  • Real-World Manipulation: Evaluated on a physical Agilex-2.0 Piper manipulator across four language-conditioned tasks (move spray bottle, open drawer, open jar, sort object).

Key Insights and Ideas

  • Conservative Fusion Over Aggressive Deletion: In robotics, missing an object part due to over-filtering is far more dangerous than having a slightly noisy shape. Soft-shrinking low-support points preserves structural safety.
  • Decoupling Geometry from Appearance: Fusing only spatial coordinates while keeping appearance attributes fixed prevents the color and opacity degradation that typically plagues multi-view fusion models.
  • Failures as Constraints: A critical insight in the memory architecture is that failed trajectories shouldn't be discarded. Instead, they are explicitly tagged as negative constraints, ensuring the planner actively avoids repeating past mistakes.

Key Results

  • Higher Fidelity 3D Models: On the GSO-30 benchmark, GeoFuse-MV3D outperformed the MV-SAM3D baseline across all metrics, reducing Chamfer Distance (CD) by 2.20% and LPIPS by 2.02%, while boosting PSNR by 2.36%.
  • More Efficient Agent Planning: On Terminal-Bench and SWE-Bench Verified, the Governed KnowledgeBank improved success rates over the strong ReasoningBank baseline by 4.53% and 3.73% respectively, while consistently reducing the average number of steps (AS) and total token overhead.
  • Robust Robot Execution: In simulation (RLBench), GeneralVLA-2 successfully generated trajectories for all 14 tasks training-free, outperforming baselines like VoxPoser and Hamster. In real-world robot trials, it achieved a high success rate of 83.33% on object sorting and 53.33% on opening jars, significantly outperforming zero-shot alternatives.

Limitations and Future Work

Despite its strengths, GeneralVLA-2 remains dependent on the reliability of its upstream inputs; severe mask leakage or camera calibration drift can negatively impact the soft visual-hull calculations. Furthermore, because the geometry fusion mechanism is intentionally highly conservative, the quantitative improvements in reconstruction metrics are steady but modest.

The authors note that the current real-world validation is restricted to static tabletop setups with short horizons. They point to long-horizon mobile manipulation, handling heavy occlusions, interacting with deformable objects, and implementing interactive human-in-the-loop recovery as exciting frontiers for future work.

Summary updated

Abstract

Generalist vision-language-action systems need object-centric 3D evidence and reusable manipulation experience to plan reliable robot trajectories. GeneralVLA provides a hierarchical interface for converting language and RGB-D observations into 3D end-effector paths, but two bottlenecks remain. First, monocular SAM3D-style object reconstruction can hallucinate pose and unseen geometry, while manipulation benefits from stable object shape when calibrated multi-view observations are available. Second, the original KnowledgeBank mainly retrieves semantically similar snippets and appends new knowledge, which makes it difficult to control memory quality, conflicts, confidence, and geometric relevance. To address the first challenge, we introduce GeoFuse-MV3D, a geometry-prior-guided MV-SAM3D reconstruction branch that verifies external geometry cues with input-view masks, applies soft visual-hull support, performs axis-wise refinement, and fuses only geometry while preserving appearance. To address the second challenge, we upgrade KnowledgeBank into a governed long-term memory system with explicit quality, confidence, lifecycle, verifier, and conflict metadata, together with precision-oriented retrieval. Finally, we evaluate the reconstruction branch on GSO-30 and the memory module on Terminal-Bench 2.0 and SWE-Bench Verified; GeoFuse-MV3D improves over the MV-SAM3D baseline by reducing CD and LPIPS by 2.20% and 2.02% while increasing PSNR and SSIM by 2.36% and 1.03%, and KnowledgeBank improves over ReasoningBank by 4.53% on Terminal-Bench SR and 3.73% on SWE-Bench resolve rate, while reducing AS by 4.95% and 5.65%, respectively. Code: https://github.com/AIGeeksGroup/GeneralVLA-2. Website: https://aigeeksgroup.github.io/GeneralVLA-2.
BibTeX
@misc{wang2026generalvla,
  author = {Haoyu Wang and Guoqing Ma and Zeyu Zhang and Yandong Guo and Boxin Shi and Hao Tang},
  title = {{GeneralVLA-2: Geometry-Aware Reconstruction and Governed Memory for Robot Planning}},
  year = {2026},
  eprint = {2606.17480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.17480},
}
Figure