Object-Centric Joint Optimization for Semantic SLAM

35 concepts27 papers1 proposal

Traditional Simultaneous Localization and Mapping (SLAM) systems typically rely on low-level geometric primitives, such as points, lines, or voxels. While effective for local tracking, these primitives lack semantic meaning and are susceptible to drift, lighting changes, and occlusions. A growing body of research proposes shifting the fundamental unit of SLAM from these abstract points to high-level, recognizable objects.

By treating objects as "semantic anchors" within a global optimization framework, these systems achieve greater spatial accuracy, robust loop closure, and a more human-like understanding of the environment.

Objects as Semantic Anchors

The core intuition behind object-centric SLAM is that a recognized object—like a chair, a car, or a monitor—provides a much stronger geometric constraint than a collection of disconnected points. In a standard pose graph, nodes represent camera poses and edges represent the transformation between them. Di Stefano et al. and Salas-Moreno et al. extended this by incorporating objects as distinct vertices within the graph.

When a system recognizes a known 3D object, it establishes a "semantic anchor." Because the object’s internal geometry is assumed to be rigid and persistent, it acts as a landmark that constrains the camera's trajectory over long sequences. This approach, termed Semantic Bundle Adjustment by Di Stefano et al., allows the system to refine both the camera's path and the object's 3D position simultaneously, significantly reducing translation and rotation errors.

The Reciprocal Accuracy Boost

One of the most powerful aspects of this research direction is the synergy between camera tracking and object localization. In a joint optimization framework, improvements in one domain directly benefit the other.

  • Camera to Object: Accurate camera poses allow for better triangulation and integration of object observations across multiple views.
  • Object to Camera: Recognized objects provide high-level geometric constraints that correct camera drift.

Henning et al. demonstrated this "reciprocal accuracy boost" by jointly optimizing camera trajectories and human motion. They found that high-quality human pose estimation directly enhances camera tracking, even helping to recover absolute metric scale from a single moving camera. Similarly, Li et al. argued that processing camera poses and dynamic object trajectories within a single unified matrix provides more precise results than estimating them in separate, sequential stages.

Leveraging Shape Priors and Primitives

To handle objects without pre-existing CAD models, researchers use geometric abstractions or learned priors. Hosseinzadeh et al. integrated points, planes, and volumetric quadrics into a single factor graph, using deep learning to provide shape priors. This allows the system to enforce structural relationships, such as an object being tangent to a detected floor plane, which further stabilizes the trajectory.

For more complex geometries, Agapito et al. introduced DSP-SLAM, which uses deep shape priors (DeepSDF) to reconstruct complete 3D geometries from partial observations. By optimizing these shapes alongside camera poses in a joint bundle adjustment, the system maintains global consistency even when objects are heavily occluded. Even when specific models are unavailable, Zhang et al. showed that representing unknown objects as simple primitives like cylinders or cubes can still facilitate precise robotic interaction and mapping.

Robust Data Association and Loop Closure

A major challenge in SLAM is "data association"—determining if a current observation matches a previously seen landmark. Object-centric approaches simplify this by matching semantic identities rather than volatile visual features.

Mu et al. addressed this by treating object identity as a hidden variable within a nonparametric pose graph, allowing the system to autonomously infer the number of unique objects in a scene. For global consistency, McCormac et al. and Junfeng Yu proposed using object centroids and 3D semantic graphs for loop closure. When a previously seen object is re-identified, the system can correct large-scale trajectory drift without distorting the internal geometry of the individual object reconstructions.

This shift toward an object-based covisibility graph, as explored by Jiang et al., creates a more compact and robust representation. By anchoring the map to meaningful entities, these systems achieve a level of stability and environmental awareness that point-based methods cannot match.

Key Foundational Work

Joint Detection, Tracking and Mapping by Semantic Bundle Adjustment (2013) — contributed 2 concept elements to this direction.
SLAM++: Simultaneous Localisation and Mapping at the Level of Objects (2013) — contributed 1 concept elements to this direction.
A Versatile Learning-Based 3D Temporal Tracker: Scalable, Robust, Online (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_object_centric_pose_graph_optimization_5c32f85c,
  title = {object-centric pose graph optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-pose-graph-optimization/rdr_5c32f85c1e614b2795977eeff47dfc3a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_object_centric_pose_graph_optimization_5c32f85c_v1,
  title = {object-centric pose graph optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-pose-graph-optimization/rdr_5c32f85c1e614b2795977eeff47dfc3a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake