Have Fun Storming the Castle(s)!

Connor Anderson, Adam Teuscher, Elizabeth Anderson, Alysia Larsen, Josh Shirley, Ryan FarrellWACV 2021

TL;DR

This paper introduces the "Castles Dataset," a massive new dataset of over 770,000 images spanning more than 2,400 distinct castles worldwide. It's designed to push the boundaries of fine-grained visual recognition, instance retrieval, and geolocalization by offering a challenging domain with high intra-class variation and low inter-class similarity.

What Problem Does This Paper Address?

The field of computer vision has seen tremendous progress fueled by large-scale datasets. However, many existing datasets focus on broad categories or highly distinct objects. This leaves a gap in datasets that are both large-scale and fine-grained, meaning they contain many visually similar sub-categories. Such datasets are crucial for developing models that can distinguish between subtle differences, a skill vital for real-world applications. The authors argue that fine-grained recognition is particularly challenging due to high intra-class variation (e.g., a castle viewed from different angles, in different lighting) and low inter-class variation (e.g., castles from different countries that share similar architectural styles). They propose the Castles Dataset to address this by providing a rich, challenging domain for these tasks.

Methods

The paper introduces a novel dataset and then uses it to evaluate standard computer vision tasks.

  • Dataset Construction: The authors compiled their dataset by first searching Google Maps for castle locations across many countries. They then developed a web interface for manual filtering to remove non-castle locations and overly ruined structures. To handle duplicates, they used latitude-longitude information and calculated pairwise distances. Finally, they collected images from user-uploaded photos associated with each castle.
  • Image Pruning: A semi-automated process was used to filter images. After manually labeling a subset of images, they trained a binary classifier to identify images relevant to the exterior of castles. A conservative threshold was applied to select images, resulting in approximately 773,000 images.
  • Baseline Experiments: The authors conducted experiments on three core tasks:
  • Instance Recognition: This involves classifying an image into one of the specific castle classes. They used a ResNet-50 backbone with an extended classification head.
  • Country Prediction: This task aims to predict the country of origin for a castle image.
  • Image Retrieval: The goal is to find images of the same castle given a query image. They employed a triplet loss function to learn an embedding space.
  • Blind Castle Localization (Geolocation): This task estimates the geographic location of a castle from its visual appearance, using visual similarity to a database of geotagged images.

Datasets

  • Castles Dataset (New): This is the primary contribution. It contains over 770,000 images of more than 2,400 distinct castles from over 90 countries. Annotations include geotagged latitude/longitude, country label, approximate construction date (for 85% of classes), and a Google Maps link.
  • ImageNet: Used for pre-training the ResNet-50 backbone for various experiments.

Key Insights and Ideas

  • The "Difficulty Spectrum": The paper highlights that not all fine-grained recognition problems are equally hard. They propose that difficulty can be assessed at both the image and category level, suggesting that future research should consider this "spectrum of difficulty" rather than treating all classes as simply "different."
  • Scale and Fine-Graining: The Castles Dataset bridges the gap between large-scale landmark datasets (like Google Landmarks) and smaller, highly specialized fine-grained datasets (like bird or car datasets). It offers a large number of classes with a significant number of images per class, presenting a unique challenge.
  • Visual Similarity as a Challenge: The dataset's inherent nature—many castles looking similar across different countries or having high intra-class variation—makes it a particularly difficult benchmark for visual recognition tasks.
  • Semi-Automated Data Curation: The paper demonstrates an effective strategy for building large, specialized datasets by combining automated search with targeted manual labeling and classifier retraining for data pruning.

Key Results

  • Instance Recognition: Achieved an overall accuracy of 89.8% on the test set. However, when normalized for the number of images per class, the accuracy dropped to 82.5%, indicating that classes with fewer images are harder to recognize.
  • Country Prediction: Achieved a mean accuracy of 44.1%. Performance varied significantly by country, with Japan showing very high accuracy due to its distinctive castle style, while others like the Netherlands performed well but Austria surprisingly low.
  • Image Retrieval: The average per-class Mean Average Precision at R (MAP@R) was just under 0.1, highlighting the difficulty of this task on the dataset. More than 60% of classes had MAP@R < 0.1.
  • Blind Castle Localization: The median localization error was 594.9 km, with an average per-castle error of 989.9 km. This indicates that while the dataset is challenging, visual similarity can still provide reasonable geolocation estimates.

Limitations and Future Work

  • Data Imbalance: The dataset exhibits a class-imbalanced distribution, with some famous castles having many more images than obscure ones. This is acknowledged as a challenge for training models.
  • Image Relevance Threshold: While a semi-automated process was used for image pruning, some images that are not strictly exterior views of castles might have slipped through.
  • Future Work: The authors anticipate the dataset will be useful for a wide range of applications beyond the tested tasks, including local descriptor learning, correspondence and matching, multi-view reconstruction, and generative modeling. They also suggest further work on quantifying image difficulty.

Summary updated

BibTeX
@inproceedings{anderson2021have,
  author = {Connor Anderson and Adam Teuscher and Elizabeth Anderson and Alysia Larsen and Josh Shirley and Ryan Farrell},
  title = {{Have Fun Storming the Castle(s)!}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure