FSTs vs ICL: Generalisation in LLMs for an under-resourced language

Ximena Gutierrez-Vasques, Mikel Segura, Victor MijangosEMNLP 2025

TL;DR

This paper investigates how well Large Language Models (LLMs) can handle orthographic normalization for Otomi, an under-resourced language, comparing them to traditional Finite State Transducers (FSTs). The study reveals that while FSTs excel in zero-shot scenarios, LLMs with just a few examples (few-shot learning) significantly outperform them, highlighting the power of In-Context Learning (ICL) for low-resource languages.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown remarkable generalization capabilities, but their effectiveness in scenarios with limited prior knowledge of a language remains an open question. This is particularly critical for under-resourced languages, which often lack sufficient digital data and standardized linguistic resources. This paper tackles this challenge by examining how LLMs perform orthographic normalization for Otomi, a language group with multiple variants and limited NLP tools. Orthographic normalization is crucial for making written text consistent and usable for further language technology development. The authors aim to understand if LLMs, especially through In-Context Learning (ICL), can bridge the gap in multilingual coverage for such languages, where traditional rule-based methods like Finite State Transducers (FSTs) might struggle.

Methods

The researchers developed two main approaches for orthographic normalization of Otomi:

  • Rule-Based Method (FSTs): They built Finite State Transducers (FSTs) to convert text between different Otomi orthographic standards. This involved a two-step process: mapping source text to a phonetic alphabet (IPA) and then generating the target orthography. The rules were informed by linguistic expertise and existing documentation. This method is computationally lightweight and relies on explicit linguistic knowledge.
  • Neural Approach (LLMs with ICL): They employed LLMs (GPT-3.5 Turbo, GPT-40, LLaMA 3.1, and LLaMA 3.3) using In-Context Learning (ICL). ICL allows LLMs to learn tasks from a few provided examples (demonstrations) without parameter updates. They tested LLMs in two settings:
  • Zero-shot: The model receives the task description but no examples.
  • Few-shot: The model is provided with 10 examples of orthographic transductions.

The study specifically focused on scenarios that are challenging for FSTs, such as code-switching (mixing languages) and ambiguity in grapheme-to-phoneme mappings.

Datasets

The paper utilizes a custom dataset for evaluating orthographic normalization of Otomi.

  • Test Set: Consists of 191 sentences, totaling 4200 tokens. This dataset was specifically designed to include challenging cases like code-switching and ambiguity. The initial transduction for this set was performed using FSTs and then manually corrected.
  • Few-Shot Demonstrations: 10 representative sentences were manually selected to serve as examples for the LLMs. These were chosen to cover common transduction changes, code-switching, and ambiguity, prioritizing those with graphemes that vary across norms.

The paper also mentions that the test data was derived from a small Otomi corpus compiled from various sources and dialects.

Key Insights and Ideas

  • ICL's Power for Low-Resource Languages: The most striking finding is that LLMs, when provided with just 10 examples (few-shot learning), can significantly outperform even well-crafted FSTs for orthographic normalization in an under-resourced language. This underscores the potential of ICL to leverage limited data effectively.
  • FSTs Excel in Zero-Shot: In a zero-shot setting, where LLMs have no examples, FSTs demonstrate superior performance. This highlights the robustness of rule-based systems when explicit linguistic knowledge is encoded.
  • LLMs Struggle with Specific Target Norms: LLMs showed a notable difficulty when the target orthography was OTS, even with ICL. This is hypothesized to be due to the greater availability of digital data for the INALI norm, which represents an institutional effort, making it a more familiar target for pre-trained models.
  • Code-Switching is Not a Major Hurdle for LLMs: Surprisingly, LLMs handled code-switching very well in both zero-shot and few-shot settings, easily recognizing non-Otomi words and loanwords. This is a significant advantage over FSTs, which struggle with such mixed-language inputs.
  • Ambiguity is a Key Challenge: While LLMs improved with few-shot examples, errors in the few-shot setting often stemmed from failing to infer ambiguous transduction rules.

Key Results

  • Zero-Shot Performance: FSTs outperformed all LLMs in the zero-shot setting for orthographic normalization across various Otomi norms.
  • Few-Shot Performance: With just 10 examples, LLMs, particularly GPT-4o, significantly improved their performance, surpassing FSTs in several normalization directions, especially when the target was INALI. For instance, GPT-4o's Word Error Rate (WER) dropped from 31.5% in zero-shot to 11.2% in few-shot for the OTS→INALI task.
  • Code-Switching Handling: LLMs demonstrated strong performance in handling code-switching, a task where FSTs typically falter.
  • Target Norm Influence: LLMs struggled more when the target norm was OTS compared to INALI, even with ICL.

Limitations and Future Work

  • Limited Language and Task Scope: The study focused on Otomi orthographic normalization. The authors acknowledge that more comprehensive understanding would require testing across additional languages and tasks.
  • Exclusion of Lesser-Used Variants: While the main orthographic norms were covered, some lesser-used variants and phonological transcriptions were excluded. Phonological transcriptions were used for FSTs but not for the LLM approach.
  • Cost-Benefit Analysis: The authors suggest that practical considerations remain, such as evaluating the cost-benefit of using large models for relatively simple tasks, especially concerning accessibility for speakers and researchers.
  • Data Handling Concerns: Ethical and practical concerns regarding data handling in commercial systems need to be addressed for deployment.
  • Future Work: The paper implies that further exploration is needed to understand the nuances of LLM generalization, particularly concerning their ability to adapt to diverse linguistic phenomena and the potential for combining linguistic expertise with LLMs for improved performance in under-resourced scenarios.

Summary updated

BibTeX
@inproceedings{gutierrezvasques2025fsts,
  author = {Ximena Gutierrez-Vasques and Mikel Segura and Victor Mijangos},
  title = {{FSTs vs ICL: Generalisation in LLMs for an under-resourced language}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure