SeqPose: An End-to-End Framework to Unify Single-frame and Video-based RGB Category-Level Pose Estimation

Yuzhu Ji, Mingshan Sun, Jianyang Shi, Xiaoke Jiang, Yiqun Zhang, Haijun ZhangIJCAI 2025 · DOI

Introduction

Estimating the 6D pose (3D translation and rotation) and metric size of objects at the category level—such as a generic "chair" or "mug"—is a fundamental challenge in computer vision. While RGB-D sensors provide explicit depth information that simplifies this task, relying solely on RGB data is more cost-effective and versatile for real-world applications. However, RGB-only methods have historically struggled with depth ambiguity and the accumulation of errors across complex processing pipelines.

SeqPose introduces a paradigm shift by providing the first end-to-end framework that unifies single-frame and video-based RGB category-level pose estimation into a single-stage pipeline. By leveraging Large Vision Models (LVMs) and a novel spatial-temporal query network, SeqPose achieves state-of-the-art performance without requiring external depth sensors or multi-stage off-the-shelf detectors.

The Multi-Stage Bottleneck in Pose Estimation

Most existing category-level pose estimation methods rely on "daisy-chained" multi-stage pipelines. These typically involve an initial 2D object detector, followed by a depth estimator, and finally a non-differentiable alignment step (such as Normalized Object Coordinate Space or NOCS matching).

Comparison of traditional multi-stage pipelines versus the proposed unified SeqPose architecture.

This approach has two major flaws. First, errors accumulate at every step; if the initial 2D detection is slightly inaccurate, the final 3D pose often fails completely. Second, these methods typically treat video frames as independent images, ignoring the valuable temporal clues that could stabilize pose estimates over time. SeqPose addresses these issues by replacing multi-stage dependencies with a unified, differentiable transformer-based approach.

The SeqPose Architecture

The core of SeqPose is built upon three innovative components designed to extract and refine 3D information from 2D sequences:

  1. Hierarchical Feature Fusion: Instead of standard convolutional features, SeqPose utilizes the DINOv2 encoder to extract robust semantic features. These are fused with geometric priors—such as depth and surface normals—generated by a decoder based on DepthAnythingV2. This allows the model to "understand" the 3D structure of a scene directly from RGB input.
  2. Spatial-Temporal Sparse Query (STSQ) Network: Inspired by the DETR (Detection Transformer) architecture, SeqPose uses learnable 3D query anchors. In video mode, the model maintains a Temporal Memory Queue of queries from previous frames. This allows the network to model inter-frame correlations, effectively using the past to correct the geometry of the present. A geometry-aware cross-attention module then aggregates intra-frame features guided by the estimated depth maps.
  3. 9-DoF Prediction Head: The model directly regresses the 3D translation, 3D rotation (using a stable 6D continuous representation), and metric size. It employs bipartite matching to pair predicted queries with ground-truth objects during training, ensuring a fully end-to-end learning process.
Detailed architecture of SeqPose, highlighting hierarchical feature fusion, the STSQ network, and the pose prediction head.

Experimental Results and Key Insights

SeqPose was evaluated on the REAL275 and CAMERA25 benchmarks, where it demonstrated significant performance gains over previous state-of-the-art (SOTA) RGB-only methods.

  • Accuracy: On the REAL275 dataset, SeqPose improved the mAP@10cm (mean Average Precision within 10cm translation error) by 26.0% compared to the previous leader, DMSR.
  • Orientation: It achieved a 5.4% gain in mAP@10°, showcasing superior precision in estimating object rotation.
  • Efficiency: Despite its sophisticated architecture, the model maintains a near-real-time inference speed of 6.2 FPS on a single consumer-grade GPU (RTX 4090).

Key insights from the study reveal that Large Vision Models like DINOv2 provide internal representations rich enough to replace explicit 3D templates used in older methods. Furthermore, the authors demonstrate that temporal consistency acts as a powerful geometric constraint, helping the model resolve depth ambiguities that are impossible to solve from a single RGB frame alone.

Conclusion and Future Work

SeqPose represents a significant advancement in unifying single-frame and video-based pose estimation. By eliminating the need for multi-stage pipelines and external depth sensors, it provides a more robust and efficient solution for category-level 6D pose estimation.

One remaining challenge is the handling of articulated objects, such as laptops, where the model's rigid-box assumption can lead to confusion. Future work will focus on incorporating kinematic constraints into the query network to better handle objects with moving parts and improving generalization to entirely unseen object categories.

Summary updated

BibTeX
@inproceedings{ji2025seqpose,
  author = {Yuzhu Ji and Mingshan Sun and Jianyang Shi and Xiaoke Jiang and Yiqun Zhang and Haijun Zhang},
  title = {{SeqPose: An End-to-End Framework to Unify Single-frame and Video-based RGB Category-Level Pose Estimation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/137},
}
Figure