Analyzing Finetuned Vision Models for Mixtec Codex Interpretation

Alexander Webber, Zachary Sayers, Amy Wu, Elizabeth Thorner, Justin Witter, Gabriel Ayoubi, Christan GrantNAACL 2024

TL;DR

This paper explores the application of finetuned deep learning models (VGG-16 and ViT-16) for interpreting figures within historical Mixtec codices, a low-resource domain. By creating a new labeled dataset and evaluating model performance, the authors demonstrate that these models can accurately classify gender and pose, offering a computational approach to understanding these complex visual narratives.

What Problem Does This Paper Address?

Historical pictographic systems, like the Mixtec codices, offer invaluable insights into past cultures, events, and beliefs. However, their interpretation is often challenging due to their complexity, the degradation of materials over time, and the specialized knowledge required for decipherment. Mixtec codices, in particular, are highly structured, with specific conventions for depicting figures that convey narrative information. This paper addresses the gap in computational tools for analyzing these low-resource, visually rich historical documents, aiming to assist researchers and domain experts in understanding their content.

Methods

The researchers employed a two-pronged approach using established deep learning architectures:

  • Figure Segmentation: They utilized the Segment Anything Model (SAM) to automatically extract individual figures from the codices. This is a standard but powerful technique for object detection and segmentation in images.
  • Finetuning Vision Models: Two popular vision models, VGG-16 (a Convolutional Neural Network) and ViT-16 (a Vision Transformer), were finetuned. Finetuning involves taking a model pre-trained on a large, general dataset (like ImageNet) and further training it on a smaller, specific dataset to adapt it to a new task. This is a common practice for achieving good performance on specialized tasks with limited data.
  • Binary Classification Tasks: The finetuned models were trained to perform two binary classification tasks:
  • Gender: Distinguishing between male and female figures, based on established Mixtec iconographic conventions (e.g., loincloths for men, dresses for women).
  • Pose: Classifying figures as either standing or not standing.
  • Feature Visualization: To understand what features the models learned, they generated class activation maps and attention maps. This is a standard technique for interpreting the "black box" of deep learning models, helping to see which parts of an image the model focuses on.

Datasets

The study introduced a novel dataset derived from three Mixtec codices:

  • Codex Vindobonensis Mexicanus I: Contains mythological and historical founding of Mixtec kingdoms.
  • Codex Selden: Focuses on the founding of the kingdom of Jaltepec and its ruler, Lady 6 Monkey.
  • Codex Zouche-Nuttall: Primarily illustrates the life and conquests of Lord 8 Deer Jaguar Claw, along with ancestral histories.

Approximately 1300 figures were extracted and labeled for gender and pose. The dataset was split into training (60%), testing (20%), and validation (20%) sets. Data augmentation techniques like random flips and blocking were used to mitigate overfitting due to the limited dataset size.

Key Insights and Ideas

  • ViT's Robustness to Learning Rates: While both VGG and ViT achieved high accuracy, ViT-16 demonstrated more consistent performance across a wider range of learning rates, making it potentially more reliable and easier to tune.
  • Model Alignment with Expert Knowledge: The feature visualization techniques (attention maps) showed that the models learned to focus on features that domain experts identify as crucial for distinguishing gender (e.g., loincloths, skirts, hair styles). This suggests that computational models can indeed capture culturally relevant visual cues.
  • Impact of "Blocking" on Model Performance: When parts of the defining features of figures were intentionally "blocked" (masked) in reference images, the models' accuracy decreased. This further validates that the models were learning from these specific, expert-identified features.
  • Potential for Low-Resource Domains: The success of finetuning established models on a relatively small, specialized dataset highlights the potential for applying these techniques to other low-resource historical or cultural domains.

Key Results

  • Both VGG-16 and ViT-16 achieved high testing accuracies, around 97.8% for gender and 97.8% (VGG) / 97.4% (ViT) for pose, with small standard deviations.
  • ViT-16 generally outperformed VGG-16, especially at higher learning rates.
  • When reference images had defining features blocked, ViT-16's accuracy dropped from 100% (unblocked) to 79% (singly blocked) and 63% (doubly blocked), indicating reliance on these specific features.

Limitations and Future Work

  • Low-Resource Nature: The primary limitation is the small size of the available Mixtec codices, which restricts the scale of the dataset.
  • Pretrained Model Biases: The models inherit biases from their original training data, which are not specific to the Mixtec domain. The impact of these inherited biases on downstream tasks is unknown.
  • Expert Disagreement: The interpretation of certain scenes in Mixtec codices is subject to scholarly debate, which can complicate labeling and evaluation.
  • Future Work: The authors suggest extending this work to classify segmented figures within a scene, analyze relationships between figures, and ultimately extract narratives from codices. They also mention exploring more environmentally efficient models and collaborating with Mixtec domain experts and native speakers for continued development.

Summary updated

BibTeX
@inproceedings{webber2024analyzing,
  author = {Alexander Webber and Zachary Sayers and Amy Wu and Elizabeth Thorner and Justin Witter and Gabriel Ayoubi and Christan Grant},
  title = {{Analyzing Finetuned Vision Models for Mixtec Codex Interpretation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure