LLMs’ morphological analyses of complex FST-generated Finnish words

Anssi Moisio, Mathias Creutz, Mikko KurimoACL 2024

TL;DR

This paper investigates whether large language models (LLMs) like GPT-4 and Llama2 can accurately perform morphological analysis on complex Finnish words. The study finds that while GPT-4 shows some capability, even state-of-the-art models struggle with Finnish morphology, suggesting they may rely on different underlying mechanisms than human language acquisition.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has led to impressive capabilities in language generation and understanding. However, a fundamental question remains: do these models truly "learn" grammar in a way comparable to humans, or do they simply master statistical patterns without deep grammatical understanding? This paper delves into this debate by focusing on Finnish, a morphologically rich language, and specifically tests LLMs' ability to analyze complex noun forms. The challenge lies in determining if LLMs can generalize grammatical rules beyond what they've explicitly seen in training data, a crucial aspect of human language acquisition.

Methods

The researchers employed a novel approach by directly asking LLMs to perform a morphological classification task. Instead of relying on indirect methods like probing neural network representations or analyzing next-token predictions, they presented LLMs with Finnish inflected nouns and asked them to identify the base form, number, grammatical case, and possessive suffix. This direct questioning method aims to provide a more unambiguous evaluation.

To generate the test data, they used a Finite-State Transducer (FST) tool called Omorfi to create a large set of complex, inflected Finnish noun forms. This approach ensures that the generated words are unlikely to have been present in the LLMs' training data, thus requiring genuine morphological generalization. The LLMs were tested with varying numbers of examples in the prompt (0, 1, 5, and 10 shots) to assess the impact of few-shot learning. For comparison, simple recurrent neural networks (RNNs) were also trained on the same task.

Datasets

The study generated its own dataset of complex Finnish noun forms using the Omorfi tool. This dataset comprises approximately 25 million word forms, from which a random sample of 2000 inflected nouns was used as the test set. The researchers deliberately avoided existing Finnish morphology datasets like MorphyNet and UniMorph to prevent potential data contamination from LLM training data and to focus on generating more complex forms that require generalization.

Key Insights and Ideas

  • LLMs don't learn like humans: The results strongly suggest that LLMs, even advanced ones like GPT-4, do not acquire Finnish morphology in the same systematic, rule-based way humans do.
  • Generalization is key: The use of FST-generated complex words highlights the LLMs' struggle with morphological generalization, a hallmark of human language learning.
  • Tokenization matters: The paper hints that subword tokenization methods (like BPE) used by LLMs might hinder their ability to process morphology effectively, as multiple embeddings might be needed for a single morpheme.
  • Implicit vs. Explicit Knowledge: While LLMs can generate grammatically plausible Finnish words, their performance on explicit morphological analysis tasks reveals a gap in their understanding.

Key Results

  • GPT-4 shows promise but falters: GPT-4-turbo achieved the highest accuracy among the LLMs, but still performed poorly on the task, especially with complex forms. Its accuracy was not close to perfect.
  • GPT-3.5 struggles significantly: GPT-3.5-turbo performed worse than GPT-4, indicating a substantial gap between these models.
  • Smaller models fail: Llama2-70B and Poro-34B performed very poorly, failing nearly completely on the task.
  • RNNs outperform LLMs: Simple RNNs trained on the same task, even with limited data, achieved higher accuracies than the LLMs, suggesting that explicit training on morphological tasks can be more effective for this specific problem.
  • Case and possessive suffix analysis are challenging: Confusion matrices revealed that LLMs frequently misclassified grammatical cases and possessive suffixes, particularly with more complex word forms.

Limitations and Future Work

The authors acknowledge several limitations:

  • Single Language: The study was conducted solely on Finnish, limiting generalizations to other languages.
  • Limited LLMs: Only four LLMs were tested, and the performance on other models might differ.
  • Prompt Optimization: The prompts were not extensively optimized, suggesting that better prompting might elicit improved performance.
  • Implicit Knowledge: The study focuses on explicit metalinguistic tasks. It's possible LLMs possess some implicit morphological knowledge that isn't captured by these tests.

Future work could involve testing LLMs on a wider range of languages and exploring more sophisticated prompting strategies. Further investigation into how tokenization impacts morphological understanding would also be valuable.

Summary updated

BibTeX
@inproceedings{moisio2024llms,
  author = {Anssi Moisio and Mathias Creutz and Mikko Kurimo},
  title = {{LLMs’ morphological analyses of complex FST-generated Finnish words}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure