Compositionality of Complex Graphemes in the Undeciphered Proto-Elamite Script using Image and Text Embedding Models

Logan Born, Kathryn Kelley, M. Willis Monroe, Anoop SarkarACL 2021

TL;DR

This paper introduces an image-aware language modeling architecture to analyze the undeciphered Proto-Elamite script. By treating signs as images, they uncover evidence of semantic compositionality in complex graphemes (combinations of signs), suggesting a structured system beyond mere visual association. This approach offers a novel way to analyze ancient scripts, abstracting away from human annotation biases.

What Problem Does This Paper Address?

The Proto-Elamite (PE) script, one of the world's earliest writing systems, remains largely undeciphered, with its underlying language unknown. While some numerical systems and sign usage rules are understood, the semantic meaning of many signs, particularly complex graphemes (CGs) formed by combining simpler signs, is unclear. Understanding the compositionality of these CGs is crucial for decipherment, as it can reveal how meaning is constructed even without knowing the full language. Previous approaches often rely on transliterated sign names, which might obscure underlying visual or semantic relationships. This paper aims to bridge this gap by using multimodal embeddings to investigate the compositional nature of CGs in PE.

Methods

The core of this work is a novel image-aware language modeling architecture. This architecture combines two embedding components:

  1. Text-based embeddings: Standard embeddings for sign names (similar to word embeddings).
  2. Image-based embeddings: A Convolutional Neural Network (CNN) processes images of the signs to extract visual features, which are then embedded.

These two embeddings are concatenated and fed into a Bidirectional Long Short-Term Memory (BiLSTM) network. This model is trained to predict the next sign in a sequence. Crucially, the authors also train an image-only model (using only the CNN embeddings) and a text-only model (using only the text embeddings) to disentangle the contributions of visual information versus textual labels.

They evaluate compositionality using two main metrics:

  • Additive Compositionality: They hypothesize that if a CG is semantically compositional, its embedding should be approximately the sum of the embeddings of its constituent signs. They measure how often this holds true.
  • Pairing Consistency Score (PCS): This metric assesses whether the relationships between signs within CGs are consistent across different combinations, indicating predictable semantic contributions.
  • Analogies: They search for analogical relationships of the form A:B::C:D within the embedding space, particularly for CGs sharing components.

Datasets

The study uses the Proto-Elamite corpus from Born et al. (2019), which is a cleaned version of texts originally published by the Cuneiform Digital Library Initiative (CDLI). This corpus contains digitized transliterations from 1399 tablets, totaling 11013 lines and 33778 tokens. Approximately 1107 tokens (nearly half the sign types) are labeled as CGs. The dataset is split into training and validation sets, with rare signs (occurring 3 or fewer times) replaced by an "UNK" token.

Key Insights and Ideas

  • Sign names can be distractors: The image-aware language model (LM) consistently outperforms the text-only LM and even the image-recognition-only model in tasks related to compositionality and analogy. This suggests that the human-assigned sign names, while useful for annotation, can sometimes obscure the underlying visual and distributional patterns that an image-aware model can capture.
  • Proto-Elamite CGs show semantic compositionality: The study provides strong evidence that many complex graphemes in Proto-Elamite are not purely idiomatic but have meanings that are at least partly compositional, meaning their meaning is derived from the meanings of their constituent signs.
  • A hierarchy of sign containment exists: The research uncovers a "grammar" or partial ordering governing how signs can be combined into CGs, suggesting that outer signs are generally "higher" in the hierarchy than inner signs. This structure is a novel finding for PE scholarship.
  • Inner signs have a more consistent role: The Pairing Consistency Score (PCS) reveals that inner signs tend to have a more consistent and predictable impact on the meaning of CGs compared to outer signs.

Key Results

  • Compositionality Detection: The image-only LM identified compositional CGs significantly more often than text-only models or even a dedicated image recognition model. For instance, at a similarity cutoff of k=15, the image LM found compositional representations for 61 CGs, compared to only 1 for the text-only LM.
  • Analogies: The image LM found a substantially larger number of meaningful analogies between CGs sharing components than text-only models. For k=15, the image LM identified 521 such analogies, compared to 21 for the text-only LM.
  • PCS Scores: The image-aware models (lm.image+text and lm.image) achieved the highest PCS scores for inner signs (0.719 and 0.760 respectively), indicating their consistent contribution to CG meaning.

Limitations and Future Work

  • Data Sparsity: The rarity of most PE signs is a significant challenge, making it difficult for models to learn robust representations. The authors address this by replacing rare signs with "UNK" tokens, but it remains a limitation.
  • Human Annotation Bias: The reliance on existing transliterations and sign labels, created by human annotators, can introduce biases or obscure true relationships. The authors suggest future work could benefit from relabeling signs based on a combination of context and sign shape.
  • Future Work: The authors propose several avenues:
  • Relabeling signs: To address potential biases and improve data sparsity.
  • Cross-script comparison: Training embedding models on proto-cuneiform, a structurally similar script with known meanings, could provide valuable comparative insights.
  • Exploring the discovered hierarchy: Further investigation into the ordering of signs in CGs could reveal deeper administrative or semantic structures.
  • Applying to other undeciphered scripts: The image-aware architecture is presented as a generalizable tool for decipherment.

Summary updated

BibTeX
@inproceedings{born2021compositionality,
  author = {Logan Born and Kathryn Kelley and M. Willis Monroe and Anoop Sarkar},
  title = {{Compositionality of Complex Graphemes in the Undeciphered Proto-Elamite Script using Image and Text Embedding Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure