Cross-Paced Representation Learning with Partial Curricula for Sketch-based Image Retrieval

Dan Xu, Xavier Alameda-Pineda, Jingkuan Song, Elisa Ricci, Nicu Sebe

Abstract

In this paper we address the problem of learning robust cross-domain representations for sketch-based image retrieval (SBIR). While most SBIR approaches focus on extracting low- and mid-level descriptors for direct feature matching, recent works have shown the benefit of learning coupled feature representations to describe data from two related sources. However, cross-domain representation learning methods are typically cast into non-convex minimization problems that are difficult to optimize, leading to unsatisfactory performance. Inspired by self-paced learning, a learning methodology designed to overcome convergence issues related to local optima by exploiting the samples in a meaningful order (i.e. easy to hard), we introduce the cross-paced partial curriculum learning (CPPCL) framework. Compared with existing self-paced learning methods which only consider a single modality and cannot deal with prior knowledge, CPPCL is specifically designed to assess the learning pace by jointly handling data from dual sources and modality-specific prior information provided in the form of partial curricula. Additionally, thanks to the learned dictionaries, we demonstrate that the proposed CPPCL embeds robust coupled representations for SBIR. Our approach is extensively evaluated on four publicly available datasets (i.e. CUFS, Flickr15K, QueenMary SBIR and TU-Berlin Extension datasets), showing superior performance over competing SBIR methods.

Introduction

Sketch-based Image Retrieval (SBIR) is a fundamental task in computer vision that requires matching sparse, iconic line drawings (sketches) with dense, colorful photographs (images). The primary challenge lies in the significant domain gap between these two modalities. To bridge this gap, researchers typically attempt to learn a shared feature space where semantically similar objects from both domains are represented by similar mathematical vectors.

However, the optimization of these cross-domain representations often involves non-convex minimization problems. Such problems are prone to getting stuck in local optima, resulting in suboptimal retrieval performance. While "Curriculum Learning" (CL) suggests training models on easy examples before moving to harder ones, traditional CL requires exhaustive manual ranking of all training data, which is impractical for large-scale datasets. Conversely, "Self-Paced Learning" (SPL) allows the model to select its own training samples based on current performance but lacks the benefit of human domain knowledge.

The CPPCL Framework

To address these limitations, we propose Cross-Paced Partial Curriculum Learning (CPPCL). This framework integrates human-provided partial rankings with the model's internal feedback to guide the training process along an "easy-to-hard" trajectory.

The concept of Cross-paced Partial Curriculum Learning, showing the transition from easy to hard samples in both sketch and image domains.

The "Cross-Paced" nature of our approach ensures that the learning pace is managed simultaneously for both the sketch and image domains. This joint evolution prevents the model from becoming an expert in one domain while remaining ineffective in the other, which is crucial for maintaining a coherent cross-domain mapping.

Methodology

The CPPCL framework is built upon a Coupled Dictionary Learning (CDL) backbone and incorporates three key elements:

  1. Partial Curricula: Instead of a full ranking, we utilize "partial" information. For images, we automate difficulty assessment using "objectness" scores—images with clear, centered objects are considered easier. For sketches, we use "edgeness" or simple human pairwise comparisons to establish relative difficulty.
  2. Self-Pacing: The model monitors its own reconstruction loss. Samples that are too difficult to represent given the current state of the model are temporarily ignored, allowing the model to solidify its understanding of simpler concepts first.
  3. Cross-Modal Regularization: We employ a Graph Laplacian regularizer to ensure that sketches and images of the same category remain close in the learned feature space.
The overall CPPCL framework architecture, integrating partial curriculum construction with coupled dictionary learning.

Experimental Evaluation

We evaluated CPPCL across four major benchmarks:

  • CUFS (CUHK Face Sketch): Focused on sketch-to-photo face recognition.
  • Flickr15K: A large-scale dataset with ~14,660 images across 33 categories.
  • Queen Mary SBIR: Features significant variations in scale and cluttered backgrounds.
  • TU-Berlin Extension: A massive dataset with 250 object categories.

Key Results

Our framework consistently outperformed state-of-the-art (SOTA) methods. On the CUFS dataset, CPPCL achieved a 98.6% recognition rate, surpassing standard CDL (97.4%). In retrieval tasks on Flickr15K, it reached a Mean Average Precision (mAP) of 0.2734, a significant improvement over the previous SOTA of approximately 0.19. Furthermore, the system maintains practical efficiency with an online retrieval time of roughly 1.18 seconds per query.

Conclusion

CPPCL effectively bridges the domain gap in SBIR by combining the strengths of curriculum learning and self-paced learning. By utilizing partial curricula, we reduce the burden of manual annotation while steering the optimization process away from poor local optima. Future work will explore end-to-end deep learning architectures to further enhance the integration of cross-paced strategies and handle more complex data relationships.

Summary updated

Abstract

In this paper we address the problem of learning robust cross-domain representations for sketch-based image retrieval (SBIR). While most SBIR approaches focus on extracting low- and mid-level descriptors for direct feature matching, recent works have shown the benefit of learning coupled feature representations to describe data from two related sources. However, cross-domain representation learning methods are typically cast into non-convex minimization problems that are difficult to optimize, leading to unsatisfactory performance. Inspired by self-paced learning, a learning methodology designed to overcome convergence issues related to local optima by exploiting the samples in a meaningful order (i.e. easy to hard), we introduce the cross-paced partial curriculum learning (CPPCL) framework. Compared with existing self-paced learning methods which only consider a single modality and cannot deal with prior knowledge, CPPCL is specifically designed to assess the learning pace by jointly handling data from dual sources and modality-specific prior information provided in the form of partial curricula. Additionally, thanks to the learned dictionaries, we demonstrate that the proposed CPPCL embeds robust coupled representations for SBIR. Our approach is extensively evaluated on four publicly available datasets (i.e. CUFS, Flickr15K, QueenMary SBIR and TU-Berlin Extension datasets), showing superior performance over competing SBIR methods.
BibTeX
@article{xu2018cross,
  author = {Dan Xu and Xavier Alameda-Pineda and Jingkuan Song and Elisa Ricci and Nicu Sebe},
  title = {{Cross-Paced Representation Learning with Partial Curricula for Sketch-based Image Retrieval}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=lhELZKL2kvs},
}
Figure