Hangul Fonts Dataset: a Hierarchical and Compositional Dataset for Interrogating Learned Representations

Jesse A. Livezey, Ahyeon Hwang, Kristofer E. Bouchard

TL;DR

This paper introduces the Hangul Fonts Dataset (HFD), a novel benchmark designed to study how deep learning models learn hierarchical and compositional structures. They found that while supervised models can partially capture the geometric hierarchy, neither supervised nor unsupervised models effectively learn the compositional structure of the data.

What Problem Does This Paper Address?

Understanding how deep neural networks learn and represent hierarchical and compositional structures is crucial for advancing representation learning and applying these models in interpretable ways, especially in scientific domains. However, existing benchmark datasets often lack well-defined hierarchical or compositional structures, or this structure is unknown, making it difficult to precisely analyze learned representations. This gap hinders the development of new methods capable of learning these complex properties.

Methods

The authors introduce the Hangul Fonts Dataset (HFD), which is built upon the Korean writing system (Hangul). Each Hangul syllable (block) is composed of initial, medial, and final consonant/vowel glyphs. The dataset leverages the inherent structure of Hangul to provide ground-truth annotations for:

  • Hierarchical Structure: Based on the geometric layout of glyphs within a syllable (e.g., placement of initial, medial, and final components).
  • Compositional Structure: Defined by the atomic glyphs used and their variations (rotations, translations, scalings) that form the individual components.

They then evaluate various machine learning methods, including:

  • Shallow Unsupervised Methods: Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Non-negative Matrix Factorization (NMF).
  • Deep Unsupervised Methods: Variational Autoencoders (VAEs) and Beta-VAEs (β-VAEs).
  • Deep Supervised Methods: Fully-connected networks trained to predict initial, medial, or final components.

The evaluation focuses on how well these methods' learned representations capture the known hierarchical and compositional structures of the HFD, using techniques like k-means clustering and logistic regression for decoding.

Datasets

  • Hangul Fonts Dataset (HFD): A newly introduced benchmark dataset.
  • Composition: 35 fonts of the Korean writing system (Hangul).
  • Instances: 11,172 unique blocks (syllables) per font, totaling 391,020 annotated images.
  • Structure: Ground-truth annotations for initial, medial, and final components, geometric hierarchy, and compositional "bag-of-atoms" features.
  • Variations: Includes naturalistic style variations across fonts, as well as rotations, translations, and scalings of glyphs.

Key Insights and Ideas

  • The HFD as a Unique Benchmark: The HFD is valuable because it provides a dataset with known hierarchical and compositional structure, combined with naturalistic variations, which is a significant gap in current benchmarks.
  • Supervised Models Learn Hierarchy, Not Composition: Supervised deep networks show a notable ability to learn representations that reflect the geometric hierarchy of Hangul characters, but they struggle to capture the compositional aspect (how glyphs are formed from smaller parts).
  • Unsupervised Models Lag Behind: Both shallow and deep unsupervised methods demonstrate only modest evidence of learning either the hierarchical or compositional structure, suggesting they are not well-suited for extracting these complex relationships from this type of data.
  • Interpretability Challenges: The findings highlight the ongoing challenge of understanding what specific structures deep networks learn, especially when the goal is to extract latent generative variables beyond simple classification.

Key Results

  • Supervised Networks: Achieved high classification accuracies (e.g., >97%) for predicting initial, medial, and final components. Their representations showed marginal recovery of the "all_geometry" variable and some interaction with medial geometry when predicting initial components.
  • Unsupervised Methods (PCA, ICA, NMF, VAEs, β-VAEs): Showed very limited success in recovering hierarchical or compositional variables. Normalized clustering accuracies for geometry variables were generally below 0.25, and often close to 0 for unsupervised methods.
  • Compositional Features: The "bag-of-atoms" compositional features were poorly predicted by all tested methods, often performing at or below chance.
  • Representation Localization: Logistic regression analysis suggested that unsupervised VAEs learn distributed representations, while supervised networks tend to learn more localized representations for the variables they are trained on.

Limitations and Future Work

  • Limited Variation in Fonts: While the HFD includes variations across fonts, the authors acknowledge that the amount of variation might be smaller than in handwritten datasets.
  • Encoding Compositionality: The "bag-of-atoms" encoding for compositionality ignores potentially useful structural information, similar to how "bag-of-words" discards word order in NLP.
  • Network Size: The study primarily used relatively small fully-connected and convolutional networks. Future work could explore larger and more complex architectures (e.g., RNNs, transformers).
  • Evaluation Metrics: There's a need for better unsupervised or semi-supervised metrics to evaluate structure recovery tasks like disentangling, hierarchy, and compositionality.
  • Future Directions: The authors suggest extending the dataset with more fonts, exploring different network architectures, and developing new methods specifically designed to extract hierarchical and compositional structures. They also emphasize the importance of developing methods for hypothesis testing in scientific domains using deep learning representations.

Summary updated

Abstract

Hierarchy and compositionality are common latent properties in many natural and scientific datasets. Determining when a deep network's hidden activations represent hierarchy and compositionality is important both for understanding deep representation learning and for applying deep networks in domains where interpretability is crucial. However, current benchmark machine learning datasets either have little hierarchical or compositional structure, or the structure is not known. This gap impedes precise analysis of a network's representations and thus hinders development of new methods that can learn such properties. To address this gap, we developed a new benchmark dataset with known hierarchical and compositional structure. The Hangul Fonts Dataset (HFD) is comprised of 35 fonts from the Korean writing system (Hangul), each with 11,172 blocks (syllables) composed from the product of initial consonant, medial vowel, and final consonant glyphs. All blocks can be grouped into a few geometric types which induces a hierarchy across blocks. In addition, each block is composed of individual glyphs with rotations, translations, scalings, and naturalistic style variation across fonts. We find that both shallow and deep unsupervised methods only show modest evidence of hierarchy and compositionality in their representations of the HFD compared to supervised deep networks. Supervised deep network representations contain structure related to the geometrical hierarchy of the characters, but the compositional structure of the data is not evident. Thus, HFD enables the identification of shortcomings in existing methods, a critical first step toward developing new machine learning algorithms to extract hierarchical and compositional structure in the context of naturalistic variability.
BibTeX
@inproceedings{livezey2022hangul,
  author = {Jesse A. Livezey and Ahyeon Hwang and Kristofer E. Bouchard},
  title = {{Hangul Fonts Dataset: a Hierarchical and Compositional Dataset for Interrogating Learned Representations}},
  booktitle = {ICIAP},
  year = {2022},
  doi = {10.1007/978-3-031-06433-3\_1},
}
Figure