Cross-Paced Representation Learning With Partial Curricula for Sketch-Based Image Retrieval

Dan Xu, Xavier Alameda-Pineda, Jingkuan Song, Elisa Ricci, Nicu Sebe

TL;DR

This paper introduces Cross-Paced Partial Curriculum Learning (CPPCL), a framework that improves sketch-based image retrieval by training models on an "easy-to-hard" trajectory. By combining human-provided partial rankings with the model's own feedback, the authors achieve state-of-the-art results across multiple benchmarks, effectively bridging the domain gap between sparse sketches and complex natural images.

What Problem Does This Paper Address?

Sketch-based Image Retrieval (SBIR) is inherently difficult because it requires a computer to match two very different things: a sparse, iconic line drawing (the sketch) and a dense, colorful photograph (the image).

To solve this, researchers usually try to learn a "shared space" where a sketch of a cat and a photo of a cat have similar mathematical representations. However, the math behind this is "non-convex," meaning the optimization process is like trying to find the lowest point in a jagged mountain range in the dark—it is incredibly easy for the model to get stuck in a "local optimum" (a shallow valley) rather than finding the best possible solution.

Traditional "Curriculum Learning" (CL) helps by showing the model easy examples first, but it requires a human to rank every single image by difficulty—a task that doesn't scale. "Self-Paced Learning" (SPL) lets the model choose its own easy examples, but it lacks human intuition. This paper seeks to combine the best of both worlds for the specific challenge of cross-domain retrieval.

Methods

The authors propose CPPCL, which integrates three main components into a Coupled Dictionary Learning (CDL) framework:

  1. Partial Curricula: Instead of requiring a full ranking of thousands of images, the authors use "partial" information. For images, they use an automated "objectness" score (less cluttered images are "easier"). For sketches, they use "edgeness" or simple human pairwise comparisons (e.g., "Sketch A is simpler than Sketch B").
  2. Self-Pacing: The model looks at its own reconstruction loss. If it finds a sample too hard to represent given its current knowledge, it ignores that sample until later iterations.
  3. Cross-Modal Regularization: A Graph Laplacian regularizer ensures that if a sketch and an image belong to the same category, their representations stay close together in the learned feature space.

The "Cross-Paced" aspect is key: the model simultaneously manages the learning pace for both the sketch domain and the image domain, ensuring they evolve together.

Datasets

The approach was evaluated on four major benchmarks:

  • CUFS (CUHK Face Sketch): A dataset of 188 face-photo pairs used for sketch-to-photo recognition.
  • Flickr15K: A large-scale dataset with ~14,660 images and 330 sketches across 33 categories.
  • Queen Mary SBIR: A more complex dataset featuring significant scale variations and cluttered backgrounds.
  • TU-Berlin Extension: A massive dataset with 250 object categories and over 200,000 natural images.

Key Insights and Ideas

  • The Power of "Partial": You don't need a perfect teacher to guide a model. Providing just a few "this is easier than that" constraints (partial curricula) is enough to steer the self-paced learning process away from bad local optima.
  • Objectness as Difficulty: One of the "aha" moments is using "objectness" heatmaps to automate image difficulty. An image where the object is clear and centered is objectively easier for a model to correlate with a sketch than a tiny object in a busy background.
  • Joint Evolution: By pacing the learning of sketches and images together, the model avoids a situation where it becomes an "expert" in one domain while remaining "clueless" in the other, which would break the cross-domain mapping.

Key Results

The CPPCL framework consistently outperformed existing state-of-the-art (SOTA) methods:

  • Accuracy: On the CUFS face dataset, it reached a 98.6% recognition rate, beating standard Coupled Dictionary Learning (97.4%).
  • Retrieval Performance: On Flickr15K, it achieved a Mean Average Precision (mAP) of 0.2734, a massive jump over the previous SOTA of ~0.19.
  • Efficiency: Despite the complex training process, the online retrieval time is roughly 1.18 seconds per query, making it practical for real-world applications.
  • Qualitative Success: The model proved particularly robust at retrieving the correct objects even when the query sketches were highly simplified or the target images had significant background noise.

Limitations and Future Work

While the "partial" curriculum reduces human effort, the sketch domain still benefits significantly from manual annotations, which are harder to automate than image "objectness." The authors note that their current model uses mid-level features (like LKS) and CNN features; they suggest that moving toward a fully end-to-end deep learning architecture that incorporates this cross-paced strategy could yield even higher performance. They also point toward exploring "deep structured learning" to better handle the complex relationships within the data.

Summary updated

Abstract

In this paper, we address the problem of learning robust cross-domain representations for sketch-based image retrieval (SBIR). While, most SBIR approaches focus on extracting low- and mid-level descriptors for direct feature matching, recent works have shown the benefit of learning coupled feature representations to describe data from two related sources. However, cross-domain representation learning methods are typically cast into non-convex minimization problems that are difficult to optimize, leading to unsatisfactory performance. Inspired by self-paced learning (SPL), a learning methodology designed to overcome convergence issues related to local optima by exploiting the samples in a meaningful order (i.e., easy to hard), we introduce the cross-paced partial curriculum learning (CPPCL) framework. Compared with existing SPL methods which only consider a single modality and cannot deal with prior knowledge, CPPCL is specifically designed to assess the learning pace by jointly handling data from dual sources and modality-specific prior information provided in the form of partial curricula. In addition, thanks to the learned dictionaries, we demonstrate that the proposed CPPCL embeds robust coupled representations for SBIR. Our approach is extensively evaluated on four publicly available datasets (i.e., CUFS, Flickr15K, QueenMary SBIR, and TU-Berlin Extension datasets), showing superior performance over competing SBIR methods.
BibTeX
@article{xu2018cross,
  author = {Dan Xu and Xavier Alameda-Pineda and Jingkuan Song and Elisa Ricci and Nicu Sebe},
  title = {{Cross-Paced Representation Learning With Partial Curricula for Sketch-Based Image Retrieval}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=lhELZKL2kvs},
}

Loading related content...

Figure