RPE-PAD: Relative Pose Estimation for Pose-agnostic Anomaly Detection

Zhipeng Zhang, Mengzan Qi, Rongkang Ma, Yingying Fang, Guixu Zhang, Tieyong Zeng, Zhi Li

TL;DR

RPE-PAD introduces a query-independent framework for pose-agnostic anomaly detection, moving away from slow, per-image optimization. By utilizing an iterative relative pose refinement network and a novel view synthesis scheme, it achieves state-of-the-art accuracy on real-world datasets while delivering a massive 373x speedup in inference time compared to previous methods.

What Problem Does This Paper Address?

In industrial inspection, anomaly detection (AD) usually assumes that the camera is always in the same position relative to the object. However, in the real world—think of a robot arm picking up a part or a handheld scanner—the "query" image might be taken from an entirely different angle than the "normal" images seen during training. This is Pose-agnostic Anomaly Detection (PAD).

The current state-of-the-art (like OmniposeAD) handles this by using "query-specific" optimization. For every single new image, the system has to re-run an optimization loop (like iNeRF) to figure out the camera pose before it can look for defects. This is computationally expensive, slow, and often fails to generalize to viewpoints far from the training set. RPE-PAD aims to replace this slow optimization with a fast, feed-forward neural network that understands relative camera positions.

Methods

The authors propose a three-pronged approach to make pose estimation both fast and robust:

  1. Random View Synthesis Scheme (RVSS): To train a model to understand different angles, you need data. RPE-PAD uses 3D Gaussian Splatting (3DGS) to create a digital twin of the "normal" object. It then applies Gaussian noise to known camera poses and renders thousands of new "novel" views. This creates a rich dataset of image pairs with known relative poses.
  2. Iterative Relative Pose Refinement Network (IRPRN): Instead of guessing the pose in one shot, this network uses a hierarchical coarse-to-fine strategy. It extracts features using a ResNet backbone and then uses Dynamic Convolutions to iteratively refine the relative rotation and translation between a query image and a gallery image.
  3. Multi-Pair Training Strategy (MPTS): Rather than just comparing the query to one "best match" image, the model is trained using the top-$K$ most similar images from the gallery. This helps the network learn the 3D structure of the object more comprehensively, making it more resilient to large viewpoint shifts.

Datasets

The model was evaluated on two primary benchmarks:

  • MAD (Multi-pose Anomaly Detection): A synthetic dataset containing 20 categories of LEGO toys with simulated defects like scratches or missing pieces.
  • RAD (Real-world Anomaly Detection): A more challenging real-world collection of 13 categories, including toys and household items, featuring actual physical deformations and surface defects.

Key Insights and Ideas

  • Query-Independent is the Future: The "aha moment" here is that we don't need to optimize parameters for every new image. By training a network to understand the relationship between two images (relative pose), we can fix the model weights and achieve near-instant results.
  • Iterative Refinement Matters: Estimating 3D rotation is non-linear and difficult. By breaking the problem into "refinement layers," the model can correct its own initial coarse guesses, leading to much higher precision.
  • 3DGS as a Data Engine: The paper demonstrates that 3D Gaussian Splatting isn't just for pretty renders; it's a powerful tool for generating the massive amounts of labeled "pose-pair" data needed to train robust pose estimators.

Key Results

  • Incredible Speed: RPE-PAD performs total inference in just 0.011 seconds per image. For comparison, the previous best method (SplatPose) took 4.1 seconds, and the baseline OmniposeAD took over 58 seconds. This represents a 373x to 5300x speedup.
  • Superior Accuracy on Real Data: On the real-world RAD dataset, RPE-PAD achieved an image-level anomaly detection score (AUROC) of 77.8%, significantly outperforming SplatPose (56.7%) and OmniposeAD (47.9%).
  • Robust Generalization: Because the model learns general features of relative motion rather than memorizing specific viewpoints, it maintains high performance even when the query image comes from an angle never seen during training.

Limitations and Future Work

While RPE-PAD is a major step forward, it still relies on a high-quality 3D reconstruction (3DGS) of the "normal" object. If the initial 3D model is blurry or has artifacts, the rendered images used for comparison will be low quality, potentially leading to false positives in anomaly detection.

The authors suggest that future work could involve making the feature extraction even more robust to lighting changes and exploring end-to-end frameworks where the 3D reconstruction and pose estimation are optimized simultaneously.

Summary updated

Abstract

Pose-agnostic Anomaly Detection (PAD) aims to detect anomalies when the poses of query images are unknown and differ from those in the training set. Therefore, accurately estimating the camera poses for the query images in the test set is critical for this task. Existing query-specific framework methods require re-optimizing a new set of parameters for each query image, limiting their generalization and increasing computational burden. To overcome these limitations, we propose a novel method, Relative Pose Estimation for Pose-agnostic Anomaly Detection (RPE-PAD), which enhances both generalization and efficiency with a query-independent framework. Specifically, we propose a Random View Synthesis Scheme (RVSS) that generates new poses by adding Gaussian perturbations to the original poses, then renders the corresponding views to augment the dataset. To estimate the relative camera pose between two input images, we introduce an Iterative Relative Pose Refinement Network (IRPRN), which incorporates a hierarchical coarse-to-fine refinement strategy. Furthermore, we employ a Multi-Pair Training Strategy (MPTS) to train the proposed IRPRN, leveraging multiple image pairs to expand the relative pose transformation space during training. Extensive experiments demonstrate that our method achieves robust anomaly detection performance while significantly improving inference efficiency.
BibTeX
@inproceedings{zhang2026rpe,
  author = {Zhipeng Zhang and Mengzan Qi and Rongkang Ma and Yingying Fang and Guixu Zhang and Tieyong Zeng and Zhi Li},
  title = {{RPE-PAD: Relative Pose Estimation for Pose-agnostic Anomaly Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i15.38304},
}
Figure