How to Tell Ancient Signs Apart? Recognizing and Visualizing Maya Glyphs with CNNs

Gülcan Can, Jean-Marc Odobez, Daniel Gatica-PerezACM Journal on Computing and Cultural Heritage 2018 · DOI

TL;DR

This paper tackles the challenging task of classifying ancient Maya glyphs using Convolutional Neural Networks (CNNs), a problem hindered by limited data. They demonstrate that a sketch-specific CNN trained from scratch, leveraging techniques like batch normalization and oversampling, outperforms traditional feature extraction and fine-tuning methods, achieving over 70% accuracy. Furthermore, they visualize the learned representations to understand which parts of the glyphs are most informative, finding that these visualizations align with expert annotations and are perceived as more precise by human observers compared to those from more general-purpose CNNs.

What Problem Does This Paper Address?

The digital preservation of historical artifacts, like ancient Maya codices, opens up new avenues for quantitative analysis in the humanities. However, classifying these ancient glyphs presents a significant challenge due to the limited availability of segmented data. Traditional methods for shape recognition struggle with degraded characters and sparse shapes, necessitating the exploration of more robust techniques. This paper aims to bridge this gap by investigating the effectiveness of deep Convolutional Neural Networks (CNNs) for Maya glyph classification, specifically addressing how to train these models with limited data and how to interpret their decisions to make them useful for domain experts.

Methods

The authors explore three primary approaches for training CNNs on Maya glyph classification:

  • Feature Extraction: Using pretrained CNNs as fixed feature extractors.
  • Transfer Learning (Fine-tuning): Adapting pretrained CNNs by fine-tuning their later layers.
  • Training from Scratch: Training CNNs from the ground up using the Maya glyph data.

They compare several CNN architectures, including LeNet, Sketch-a-Net (a sketch-specific network), and Residual Networks (ResNet). To address the data scarcity, they employ techniques like oversampling with data augmentation (random geometric transformations) and batch normalization. For interpretability, they utilize gradient backpropagation and Grad-CAM to visualize the discriminative parts of the glyphs that the CNNs focus on. A crowdsourcing study was conducted to qualitatively assess the interpretability of these visualizations.

Datasets

The primary dataset used in this paper is the Maya Codical dataset, curated by crowdsourcing the segmentation of glyphs from ancient Maya codices (Dresden, Paris, Madrid). This dataset is characterized by limited samples per class and within-class variance that is relatively lower than monumental glyphs. The authors also mention using the 250-class sketch dataset for pretraining the Sketch-a-Net model and ImageNet for pretraining VGG-16 and ResNet-50.

Key Insights and Ideas

  • Sketch-Specific CNNs Shine with Limited Data: The Sketch-a-Net architecture, designed for sketch recognition, proved to be highly effective when trained from scratch on the Maya glyph data, outperforming more general-purpose networks and transfer learning approaches. This suggests that specialized architectures can be advantageous even when the target domain differs slightly from the original training domain.
  • Oversampling is Crucial for Imbalanced Datasets: For datasets with imbalanced class distributions, like the Maya glyph dataset, oversampling techniques combined with data augmentation are essential for achieving high and balanced class accuracies. This ensures that classes with fewer samples are adequately represented during training.
  • Interpretability Matters for Domain Experts: Visualizing the discriminative parts of glyphs using Grad-CAM provides valuable insights for epigraphers. The fact that these visualizations align with expert annotations suggests that CNNs can learn meaningful features that are relevant to human understanding.
  • Sketch-a-Net Visualizations are Perceived as More Precise: In a crowdsourcing study, human annotators generally found the visualizations from Sketch-a-Net to be more focused and insightful for identifying diagnostic glyph parts compared to the more diffused visualizations from ResNet-50. This preference for "focused" visualizations highlights the importance of how interpretability methods are presented to users.

Key Results

  • Classification Accuracy: The Sketch-a-Net model trained from scratch achieved 70.3% average accuracy on a challenging 150-class classification task, demonstrating its effectiveness with limited data.
  • Transfer Learning Benefits: Fine-tuning pretrained networks (VGG-16, ResNet-50) showed significant improvements over using them as feature extractors, with VGG-16 outperforming Sketch-a-Net by 8.9% to 14.8% in average classification accuracy when used as a feature extractor.
  • Visualizations Align with Experts: Grad-CAM visualizations of discriminative parts of glyphs showed good overlap with expert descriptions for a 5-glyph case study.
  • Crowdsourcing Preference: In the crowdsourcing study, Sketch-a-Net visualizations were favored by annotators for their perceived precision and focus, with 52% of glyphs being rated as better visualized by Sketch-a-Net compared to ResNet-50.

Limitations and Future Work

  • Limited Crowdsourcing Scope: The crowdsourcing study was a preliminary investigation, limited to 10 glyphs and a pilot set of 100 glyphs. The authors suggest that a more extensive study with more annotators and glyphs would be beneficial.
  • Glyph Localization Challenges: While Grad-CAM showed promise for localizing glyphs in cluttered settings (glyph-blocks), it failed in several cases, indicating room for improvement in this application.
  • Generalizability of Sketch-Specific Models: While Sketch-a-Net performed well, further research could explore its generalizability to other types of visual recognition tasks with limited data.
  • Expert Knowledge Integration: The paper highlights the potential of integrating expert knowledge more deeply into the CNN training or interpretation process.

The authors conclude that their work provides promising shape representation learning approaches for limited cultural heritage data and offers insights into the interpretability of these representations, paving the way for more automated and interpretable tools for digital humanities scholars.

Summary updated

BibTeX
@article{can2018how,
  author = {Gülcan Can and Jean-Marc Odobez and Daniel Gatica-Perez},
  title = {{How to Tell Ancient Signs Apart? Recognizing and Visualizing Maya Glyphs with CNNs}},
  journal = {ACM Journal on Computing and Cultural Heritage},
  year = {2018},
  doi = {10.1145/3230670},
}
Figure