TL;DR
LocateAnyMesh introduces a novel way to find objects in 2D images by using 3D meshes directly as queries within a Vision-Language Model (VLM). By encoding textured 3D meshes into a compact sequence of "mesh tokens," the model bypasses the need for expensive 2D rendering or view-template matching, enabling efficient and precise object localization, pointing, and description.
What Problem Does This Paper Address?
In computer vision, "grounding" refers to the task of linking a query (usually text or a 2D image) to a specific region in a scene. While text is convenient, it is often ambiguous—describing a specific green figurine among many similar ones can require a long, complex prompt.
Traditional "model-based" detection uses 3D CAD models to solve this, but it typically relies on rendering dozens of 2D views of the 3D model from different angles and matching them to the image. This is computationally heavy and view-dependent. The authors of LocateAnyMesh ask: Can we treat a 3D mesh as a first-class citizen in a VLM's prompt? Instead of rendering the mesh, they aim to feed its geometric and color features directly into the model's "thought process."
Methods
The authors propose a framework that treats 3D mesh data as a sequence of tokens, much like words in a sentence or patches in an image.
- Mesh Encoding: They sample 10,000 colored points from the surface of a 3D mesh. These points are passed through a frozen 3D encoder (OpenShape-RGB PointBERT), which compresses the complex geometry and texture into a compact sequence of 65 tokens.
- The Connector: A trainable MLP "connector" projects these 3D tokens into the same embedding space used by the VLM's language decoder.
- Two Model Variants:
- LAM-Direct: Based on the LocateAnything architecture, this model predicts bounding box coordinates directly as text tokens (e.g.,
<box><x1><y1>...).
- LAM-Direct: Based on the LocateAnything architecture, this model predicts bounding box coordinates directly as text tokens (e.g.,
- LAM-Select: Based on the VLX-Seek architecture, this model receives a set of candidate region proposals from an external 2D detector and "selects" the correct ones by outputting their IDs.
Crucially, this approach requires only a single encoder pass for a new object, and the resulting tokens can be cached for reuse across different scenes.
Datasets
The models were trained and evaluated on a mix of synthetic and real-world datasets:
- Training: Google Scanned Objects (GSO), Amazon Berkeley Objects (ABO), Objaverse (a massive 3D repository), and the BOP YCB-V dataset.
- Evaluation: The researchers tested on "held-out" (unseen) objects from HOPE, LM-O, and HomebrewedDB (HB) to measure generalization. They also used the official BOP YCB-V benchmark for standardized comparison.
Key Insights and Ideas
- Meshes as Native Language: The paper demonstrates that VLMs can learn to "understand" 3D tokens. By placing mesh tokens alongside image tokens, the model learns the spatial relationship between a 3D asset and its 2D projection in a cluttered scene.
- The Power of Color: A key "aha moment" came from an ablation study where the authors stripped color from the query meshes (making them gray). Performance plummeted (e.g., a 79.6% drop in mAP on the LM-O dataset), proving that the model relies heavily on surface texture, not just geometry, to find objects.
- Efficiency over Rendering: By avoiding the "render-and-match" loop, LocateAnyMesh offers a more streamlined pipeline for robotics or AR applications where a library of 3D assets needs to be localized in real-time.
Key Results
- Localization Accuracy:
LAM-Selectsignificantly outperformedLAM-Direct. On the HOPE dataset,LAM-Selectachieved a COCO mAP of 0.371, compared to 0.155 for the direct coordinate prediction model. - Benchmark Performance: On the official BOP YCB-V 900-image target list,
LAM-Selectreached an amodal AP of 0.7152, showing strong performance on seen objects in complex real-world images. - Inference Speed: When tested on NVIDIA Blackwell (GB200) hardware, the models achieved throughputs ranging from ~21 to 65 queries per second, depending on the variant and batch size.
- Qualitative Success: The model successfully handled "Describe" tasks, generating accurate text descriptions of 3D shapes (e.g., identifying a boot's "rounded toe box" and "block heel") without ever seeing a 2D render of the object.
Limitations and Future Work
Despite the progress, the authors highlight several areas for improvement:
- Presence Decisions: The models sometimes struggle to definitively say "this object is NOT in the image," leading to false positives.
- Identity Binding: When a scene contains multiple very similar objects (e.g., two different types of soup cans), the model can struggle to bind the specific mesh query to the exact correct instance.
- Proposal Dependency:
LAM-Selectis limited by the quality of the underlying 2D region detector. If the detector misses an object, the VLM cannot select it. - Future Directions: The authors suggest that scaling the 3D encoder and improving the "negative sampling" during training (showing the model more scenes where the queried object is absent) could resolve current weaknesses in rejection and identity precision.
Summary updated