LocateAnyMesh: Grounding 3D Mesh Queries in 2D Images with Vision-Language Models

Chris Choy, Zhiding Yu, Jan KautzarXiv 2026 · arXiv

TL;DR

LocateAnyMesh introduces a novel way to find objects in 2D images by using 3D meshes directly as queries within a Vision-Language Model (VLM). By encoding textured 3D meshes into a compact sequence of "mesh tokens," the model bypasses the need for expensive 2D rendering or view-template matching, enabling efficient and precise object localization, pointing, and description.

What Problem Does This Paper Address?

In computer vision, "grounding" refers to the task of linking a query (usually text or a 2D image) to a specific region in a scene. While text is convenient, it is often ambiguous—describing a specific green figurine among many similar ones can require a long, complex prompt.

Traditional "model-based" detection uses 3D CAD models to solve this, but it typically relies on rendering dozens of 2D views of the 3D model from different angles and matching them to the image. This is computationally heavy and view-dependent. The authors of LocateAnyMesh ask: Can we treat a 3D mesh as a first-class citizen in a VLM's prompt? Instead of rendering the mesh, they aim to feed its geometric and color features directly into the model's "thought process."

Methods

The authors propose a framework that treats 3D mesh data as a sequence of tokens, much like words in a sentence or patches in an image.

  1. Mesh Encoding: They sample 10,000 colored points from the surface of a 3D mesh. These points are passed through a frozen 3D encoder (OpenShape-RGB PointBERT), which compresses the complex geometry and texture into a compact sequence of 65 tokens.
  2. The Connector: A trainable MLP "connector" projects these 3D tokens into the same embedding space used by the VLM's language decoder.
  3. Two Model Variants:
    • LAM-Direct: Based on the LocateAnything architecture, this model predicts bounding box coordinates directly as text tokens (e.g., <box><x1><y1>...).
  4. LAM-Select: Based on the VLX-Seek architecture, this model receives a set of candidate region proposals from an external 2D detector and "selects" the correct ones by outputting their IDs.
The architecture of LAM-Select, showing how mesh, image, and text tokens are combined.

Crucially, this approach requires only a single encoder pass for a new object, and the resulting tokens can be cached for reuse across different scenes.

Datasets

The models were trained and evaluated on a mix of synthetic and real-world datasets:

  • Training: Google Scanned Objects (GSO), Amazon Berkeley Objects (ABO), Objaverse (a massive 3D repository), and the BOP YCB-V dataset.
  • Evaluation: The researchers tested on "held-out" (unseen) objects from HOPE, LM-O, and HomebrewedDB (HB) to measure generalization. They also used the official BOP YCB-V benchmark for standardized comparison.

Key Insights and Ideas

  • Meshes as Native Language: The paper demonstrates that VLMs can learn to "understand" 3D tokens. By placing mesh tokens alongside image tokens, the model learns the spatial relationship between a 3D asset and its 2D projection in a cluttered scene.
  • The Power of Color: A key "aha moment" came from an ablation study where the authors stripped color from the query meshes (making them gray). Performance plummeted (e.g., a 79.6% drop in mAP on the LM-O dataset), proving that the model relies heavily on surface texture, not just geometry, to find objects.
  • Efficiency over Rendering: By avoiding the "render-and-match" loop, LocateAnyMesh offers a more streamlined pipeline for robotics or AR applications where a library of 3D assets needs to be localized in real-time.
Figure 2: Same CAD input, different network representations. Model-based detection and LocateAnyMesh ground a supplied CAD asset in a scene image using rendered views and native 3D tokens, respectively.

Key Results

  • Localization Accuracy: LAM-Select significantly outperformed LAM-Direct. On the HOPE dataset, LAM-Select achieved a COCO mAP of 0.371, compared to 0.155 for the direct coordinate prediction model.
  • Benchmark Performance: On the official BOP YCB-V 900-image target list, LAM-Select reached an amodal AP of 0.7152, showing strong performance on seen objects in complex real-world images.
  • Inference Speed: When tested on NVIDIA Blackwell (GB200) hardware, the models achieved throughputs ranging from ~21 to 65 queries per second, depending on the variant and batch size.
  • Qualitative Success: The model successfully handled "Describe" tasks, generating accurate text descriptions of 3D shapes (e.g., identifying a boot's "rounded toe box" and "block heel") without ever seeing a 2D render of the object.

Limitations and Future Work

Despite the progress, the authors highlight several areas for improvement:

  • Presence Decisions: The models sometimes struggle to definitively say "this object is NOT in the image," leading to false positives.
  • Identity Binding: When a scene contains multiple very similar objects (e.g., two different types of soup cans), the model can struggle to bind the specific mesh query to the exact correct instance.
  • Proposal Dependency: LAM-Select is limited by the quality of the underlying 2D region detector. If the detector misses an object, the VLM cannot select it.
  • Future Directions: The authors suggest that scaling the 3D encoder and improving the "negative sampling" during training (showing the model more scenes where the queried object is absent) could resolve current weaknesses in rejection and identity precision.

Summary updated

Abstract

Textured 3D meshes specify object geometry and appearance, making them a natural query for finding a particular asset in an image. We introduce LocateAnyMesh, a vision-language model that uses the mesh itself as part of its prompt. A frozen 3D encoder maps colored surface points to a compact token sequence, which a learned connector places alongside image and instruction tokens. We evaluate two grounding models with PPAT mesh tokens: LAM-Select selects image-region proposals, while LAM-Direct predicts coordinates directly. Neither requires rendered query views or view-template matching. New objects require a single encoder pass and no object-specific fine-tuning. LAM-Select improves localization on the reported real-image suites and the official BOP YCB-V evaluation. Secondary experiments compare mesh encoders, and a LAM-Direct study isolates surface color. A single-seed LAM-Select continuation supplies pointing and mesh-only description evidence without replacing the reported localization model. The results support mesh tokens as a query representation, with remaining limitations in presence decisions, identity binding, proposal coverage, and descriptive fidelity.
BibTeX
@misc{choy2026locateanymesh,
  author = {Chris Choy and Zhiding Yu and Jan Kautz},
  title = {{LocateAnyMesh: Grounding 3D Mesh Queries in 2D Images with Vision-Language Models}},
  year = {2026},
  eprint = {2611.11692},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11692},
}
Figure