How Do Transformer-Architecture Models Address Polysemy of Korean Adverbial Postpositions?

Seongmin Mun, Guillaume DesagulierACL 2022

TL;DR

This paper investigates how Transformer models like BERT and GPT-2 handle the complex polysemy of Korean adverbial postpositions. By developing a visualization system and conducting case studies, the authors reveal that BERT generally outperforms GPT-2, and that model performance is influenced by the number of functions a postposition has, the size of its training data, and the semantic closeness of its meanings.

What Problem Does This Paper Address?

Polysemy, where a single word form has multiple related meanings or functions, is a significant challenge in natural language processing. This is particularly true for Korean adverbial postpositions, which are notoriously ambiguous, often having many-to-many associations between their form and function. While previous studies have used contextualized word embeddings to tackle this, the exact mechanisms by which models like Transformers resolve this ambiguity remain unclear. This paper aims to shed light on this by analyzing how BERT and GPT-2 interpret the polysemy of three specific Korean postpositions: -ey, -eyse, and -(u)lo.

Methods

The researchers employed two prominent Transformer-based models: BERT (specifically, KoBERT) and GPT-2 (KoGPT-2-base-v2). They developed a classification model to predict the intended function of each postposition within a sentence. A key contribution is a visualization system that allows for interactive exploration of the models' performance. This system includes:

  • Distributional Map: Visualizes sentence-level embeddings, showing how sentences with different postposition functions cluster together in a 2D space.
  • F-score Charts: Displays the classification performance (F-score) of the models over epochs, both overall and for each specific function.
  • Density-Based Clustering Graphs: Shows the number of clusters formed by the embeddings as training progresses.

The models were fine-tuned on a custom-annotated corpus of Korean sentences. For BERT, special tokens ([CLS] and [SEP]) were added to the input, while GPT-2 did not use these. The tokenization methods also differed: BERT used WordPiece, and GPT-2 used Byte Pair Encoding.

Datasets

The primary dataset used was a portion of the Sejong primary corpus, a representative corpus for Korean. Since this corpus does not directly annotate postposition functions, the researchers manually annotated 15,000 sentences (5,000 for each of the three postpositions: -ey, -eyse, and -(u)lo). This annotation was performed by three native Korean speakers, and the inter-annotator agreement was high (Fleiss' kappa scores above 0.92). The annotated data was split into training (90%) and testing (10%) sets.

Key Insights and Ideas

  • BERT's Superiority: BERT consistently outperformed GPT-2 in classifying the functions of Korean postpositions across the board. This suggests differences in their architectural design or pre-training objectives (e.g., BERT's bidirectional nature) are crucial for this task.
  • Function Count Matters: Models performed better on postpositions with fewer functions. Specifically, -eyse, with only two functions (SRC and LOC), yielded higher F-scores for both BERT and GPT-2 compared to -ey and -(u)lo, which have more functions.
  • Data Imbalance is a Hurdle: The performance of the models was significantly affected by the corpus size (frequency) of each function. Functions with fewer occurrences were harder to classify accurately.
  • Learning Over Time: Both models showed gradual improvement in performance as training epochs increased, indicating that Transformers can learn to disambiguate postposition functions with sufficient exposure.
  • Semantic Closeness is Tricky: The models struggled with identifying functions that are semantically very close to each other (e.g., DIR and LOC), even with sufficient training. This suggests that purely distributional information might not be enough to resolve such fine-grained semantic distinctions.
  • Identifying Rare Functions: BERT demonstrated an impressive ability to identify functions with low-frequency occurrences (like EFF for -(u)lo) once sufficient epochs were completed, highlighting its capacity to learn from sparse data.

Key Results

  • Overall F-scores: BERT achieved an overall F-score of 0.875 for -eyse, while GPT-2's highest overall F-score was 0.844 for -eyse. For -(u)lo, BERT achieved 0.795, and GPT-2 achieved 0.676.
  • Model Comparison: Statistical analysis (ANOVA and t-tests) confirmed significant differences in performance between BERT and GPT-2 across all postpositions.
  • Function-Specific Performance: For BERT, the highest F-scores for individual functions were: CRT (0.837) for -ey, LOC (0.93) for -eyse, and DIR (0.911) for -(u)lo. GPT-2 also showed high performance for these functions.
  • Visualization Insights: The t-SNE visualizations clearly showed how sentence embeddings for different postposition functions began to cluster as training progressed, with BERT showing more distinct clusters earlier and for rarer functions.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Postposition Scope: The study focused on only three adverbial postpositions. Future work should incorporate more postpositions with varying degrees of polysemy.
  • Semantic Closeness Challenges: The difficulty in distinguishing semantically similar functions (like DIR and LOC) remains a challenge, indicating that current models may still struggle with very nuanced semantic distinctions.
  • Data Scarcity: While BERT showed promise in identifying rare functions, the overall performance is still affected by the asymmetric distribution of functions in the data.

Future research directions suggested include:

  • Expanding Postposition Types: Investigating a wider range of Korean postpositions.
  • Exploring Other Models: Evaluating newer or different contextualized word-embedding models like GPT-3 or ELECTRA to see if they offer further advantages.
  • Improving Semantic Distinction: Developing methods to better handle semantically close functions.

Summary updated

BibTeX
@inproceedings{mun2022how,
  author = {Seongmin Mun and Guillaume Desagulier},
  title = {{How Do Transformer-Architecture Models Address Polysemy of Korean Adverbial Postpositions?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure