Fusing Semantic Knowledge with Geometric Precision for Robotic Manipulation

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 133 papers43 conceptsfrozen 2026-10-04

For a robot to pick up a mug, it needs to solve two fundamentally different problems. First, it must understand what a "mug" is—distinguishing it from a bowl or a spoon based on its appearance and purpose. Second, it must know exactly where the mug’s handle is in three-dimensional space to avoid a collision.

In robotic research, these are often treated as separate domains: semantics (the "what") and geometry (the "where"). Semantic understanding is typically driven by 2D Vision-Language Models (VLMs) trained on internet-scale data, while geometric precision comes from 3D point clouds or LiDAR. Recent work suggests that the key to generalizable manipulation lies in fusing these two modalities into a single, cohesive representation.

The Semantic-Geometric Trade-off

Traditional robotic systems often struggle with a trade-off. If they rely solely on 2D images, they benefit from the rich "common sense" of models like CLIP or DINOv2, but they lack the depth perception required for precise interaction. Conversely, systems relying purely on 3D point clouds are spatially accurate but "blind" to category-level nuances; to a point cloud, a red mug and a blue mug of the same shape are identical.

As Atalay Donat and colleagues argue in FPV-Net, relying on a single modality forces a compromise between identifying an object's identity and determining its exact coordinates. To bridge this gap, researchers like Tong Zhang have proposed the Semantic-Geometric Representation (SGR) Paper, which explicitly combines high-level semantic features from 2D models with the spatial reasoning of 3D point clouds.

Why Fusion Works: Complementary Strengths

The intuition behind this fusion is complementarity. Semantic models provide the "priors"—the general knowledge that a handle is for grasping or that a "red" object is the target. Geometric models provide the "grounding"—the specific XYZ coordinates of the surface.

  1. Generalization to Novel Objects: By using semantic embeddings from foundation models, robots can manipulate objects they have never seen before. Li Junliang and the team behind the Embodied Dexterous Grasping System (EDGS) Paper demonstrated that combining VLM reasoning with human-inspired geometric strategies allows robots to handle irregularly shaped household items with a 96.1% success rate.
  2. View Invariance: 2D images change drastically with lighting and camera angles. 3D geometry is more stable. Hongjie Fang shows in AirExo-2 that fusing dense 2D features with sparse 3D geometry creates a representation that is robust to changes in the environment, enabling zero-shot deployment.
  3. Functional Grasping: A robot shouldn't just grab an object anywhere; it needs to grab it in a way that makes it usable. Adithyavairavan Murali explored this in Same Object, Different Grasps, using a knowledge graph to provide semantic context to 3D point clouds, ensuring the robot picks up a tool by the handle rather than the business end.

Mechanisms of Integration

Simply concatenating 2D and 3D features is often insufficient. Jiazhao Zhang notes in LaDi-WM that simple fusion fails to capture the interactive relationship between an object's form and its function.

To solve this, several architectural innovations have emerged:

  • Adaptive Fusion: The Visual-Geometry Diffusion Policy (VGDP) Paper uses "modality-wise dropout" during training. This forces the network to learn from both sensors equally, preventing the model from over-relying on easy-to-process 2D features and ignoring the harder-to-parse 3D data.
  • Structured Point Maps: Xiaogang Jia introduced PointMapPolicy Paper, which projects 3D coordinates into a 2D grid. This allows the system to use efficient 2D vision architectures (like Transformers) to process geometric data, effectively "speaking the same language" as semantic models.
  • Semantic Grounding of Geometry: In SAGE, Bokui Shen uses VLMs to identify "actionable parts" (like a microwave handle) and then uses 3D perception to find the exact kinematic axis of that part. This prevents the "hallucinations" common in pure VLMs by grounding their abstract reasoning in physical reality.

From Pixels to Actions

The ultimate goal of this research direction is to move from passive perception to active manipulation. Frameworks like VO-DP Paper by Lingfeng Qian demonstrate that even with vision-only inputs, fusing semantic (DINOv2) and geometric (VGGT) priors can match the performance of expensive 3D sensors.

By treating semantics and geometry as two sides of the same coin, these systems allow robots to "imagine" the consequences of their actions in a latent space that understands both the identity of the object and the physics of the world. As Zhenyu Li suggests in SimIPU, the future of robotic foundation models lies in this seamless integration, creating agents that finally understand both the "what" and the "where" of the physical world.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fusing_semantic_knowledge_with_geometric_81fdf146,
  title = {Fusing Semantic Knowledge with Geometric Precision for Robotic Manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fusing-semantic-knowledge-with-geometric-precision-for-robotic-manipulation/rdr_81fdf14635eb4523818c5abb0261550d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_fusing_semantic_knowledge_with_geometric_81fdf146_v1,
  title = {Fusing Semantic Knowledge with Geometric Precision for Robotic Manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fusing-semantic-knowledge-with-geometric-precision-for-robotic-manipulation/rdr_81fdf14635eb4523818c5abb0261550d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}