Structure-Aware Residual-Center Representation for Self-Supervised Open-Set 3D Cross-Modal Retrieval

Yang Xu, Yifan Feng, Yu Jiang

Abstract

Existing methods of 3D cross-modal retrieval heavily lean on category distribution priors within the training set, which diminishes their efficacy when tasked with unseen categories under open-set environments. To tackle this problem, we propose the Structure-Aware Residual-Center Representation (SRCR) framework for self-supervised open-set 3D cross-modal retrieval. To address the center deviation due to category distribution differences, we utilize the Residual-Center Embedding (RCE) for each object by nested auto-encoders, rather than directly mapping them to the modality or category centers. Besides, we perform the Hierarchical Structure Learning (HSL) approach to leverage the high-order correlations among objects for generalization, by constructing a heterogeneous hypergraph structure based on hierarchical inter-modality, intra-object, and implicit-category correlations. Extensive experiments and ablation studies on four benchmarks demonstrate the superiority of our proposed framework compared to state-of-the-art methods.

Introduction

3D cross-modal retrieval (3DCMR) aims to retrieve 3D objects across different modalities, such as using a 2D image query to find a corresponding 3D point cloud. Most existing approaches operate under a "closed-set" assumption, where training and testing categories are identical. However, real-world applications frequently encounter "unseen" categories. These traditional models often struggle in "open-set" environments because they overfit to the specific category distributions of the training data, leading to center deviation—a shift in the representation space when encountering new object types. Furthermore, the reliance on labeled data limits scalability.

To address these challenges, we introduce the Structure-Aware Residual-Center Representation (SRCR) framework. SRCR is a self-supervised approach that generalizes to unseen categories by learning residual embeddings and capturing high-order structural relationships between data points.

Methodology

The SRCR framework consists of two primary modules designed to mitigate center deviation and enhance generalization: Residual-Center Embedding (RCE) and Hierarchical Structure Learning (HSL).

Overview of the SRCR framework architecture, featuring the nested auto-encoders of the RCE module and the heterogeneous hypergraph of the HSL module.

Residual-Center Embedding (RCE)

Instead of a direct mapping to fixed category centers, RCE employs nested auto-encoders. An outer auto-encoder projects basic features into a unified space. Simultaneously, an inner auto-encoder calculates a "residual"—the specific deviation of a modality-specific feature from its semantic center. This approach allows the model to preserve the object's core identity while accounting for modality-specific noise, making it more robust to the distribution shifts seen in open-set scenarios.

Hierarchical Structure Learning (HSL)

To improve generalization, the model must understand complex relationships between objects. SRCR constructs a heterogeneous hypergraph where hyperedges connect multiple vertices to capture high-order correlations. Three types of relationships are modeled:

  • Inter-modality: Correlations between different formats (e.g., image vs. point cloud) of the same category.
  • Intra-object: Correlations between different modalities of the exact same object instance.
  • Implicit-category: Correlations between an object and its k-nearest neighbors to capture latent similarities.

By performing convolutions over this hypergraph, the model learns smoothed, robust representations that are not strictly tied to training labels.

Experimental Evaluation

We evaluated SRCR on four newly established open-set benchmarks derived from public 3D datasets: OCAB (from ABO), OCNT (from NTU), OCES (from ESB), and OCMN (from ModelNet40). In these benchmarks, the testing set contains entirely different categories from the training set. Each object is represented in three modalities: multi-view images, voxels, and point clouds.

Results and Discussion

SRCR consistently outperforms state-of-the-art (SOTA) methods across all benchmarks and retrieval tasks.

Precision-Recall (PR) curves on OCAB, OCNT, OCES, and OCMN benchmarks, demonstrating the superior performance of SRCR (Ours) compared to existing methods.

On the OCNT dataset for Image-to-Point retrieval, SRCR achieved a Mean Average Precision (mAP) of 0.2861, nearly doubling the performance of the previous SOTA, HGM²R (0.1812). Similar gains were observed in Point-to-Image tasks, with improvements of 10-20% over methods like CMCL or SDML. Ablation studies confirmed that both the residual embedding strategy and the hypergraph structure are critical; removing either component leads to a significant performance drop.

The "aha moment" of this work is the shift from direct mapping to learning residuals. By modeling the deviation from the center, the framework gains the flexibility needed for open-set tasks. Furthermore, the use of hypergraphs to capture high-order "neighborhood" signals proves far more effective for generalization than simple point-to-point distance metrics.

Conclusion

The SRCR framework represents a significant advancement in open-set 3D cross-modal retrieval. By leveraging self-supervised residual learning and hierarchical structure modeling, it overcomes the limitations of category-dependent models. Future work will focus on reducing the computational complexity of hypergraph operations and exploring end-to-end training of the RCE and HSL modules to further enhance performance.

Summary updated

Abstract

Existing methods of 3D cross-modal retrieval heavily lean on category distribution priors within the training set, which diminishes their efficacy when tasked with unseen categories under open-set environments. To tackle this problem, we propose the Structure-Aware Residual-Center Representation (SRCR) framework for self-supervised open-set 3D cross-modal retrieval. To address the center deviation due to category distribution differences, we utilize the Residual-Center Embedding (RCE) for each object by nested auto-encoders, rather than directly mapping them to the modality or category centers. Besides, we perform the Hierarchical Structure Learning (HSL) approach to leverage the high-order correlations among objects for generalization, by constructing a heterogeneous hypergraph structure based on hierarchical inter-modality, intra-object, and implicit-category correlations. Extensive experiments and ablation studies on four benchmarks demonstrate the superiority of our proposed framework compared to state-of-the-art methods.
BibTeX
@inproceedings{xu2024structure,
  author = {Yang Xu and Yifan Feng and Yu Jiang},
  title = {{Structure-Aware Residual-Center Representation for Self-Supervised Open-Set 3D Cross-Modal Retrieval}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10687928},
}
Figure