Introduction
Historical and domain-specific texts frequently employ abbreviations to conserve space. In the Slovenian Biographical Lexicon (SBL), terms like "b." (born) or "gr." (graduated) are ubiquitous. While human readers can easily infer their meaning, these abbreviations pose significant challenges for modern Natural Language Processing (NLP) pipelines. They often trigger tokenization errors—where a period is misinterpreted as a sentence boundary—and lead to information loss because the abbreviated forms are typically Out-of-Vocabulary (OOV) for standard models. This paper presents a robust method using the SloBERTa language model to identify and expand these abbreviations, significantly improving downstream tasks like Named Entity Recognition (NER).
Methodology: A Two-Step Pipeline
The authors propose a specialized pipeline leveraging the transformer-based SloBERTa model, which is pre-trained specifically for the Slovenian language.
- Abbreviation Identification: To circumvent fragile standard tokenizers, the text is first split by whitespace into "dirty tokens." A binary classifier fine-tuned on SloBERTa then distinguishes between regular words and abbreviations.
- Abbreviation Expansion: Identified abbreviations are processed using SloBERTa’s Masked Language Modeling (MLM) capability. The abbreviation is replaced with a
<mask>token, and the model predicts the most likely word based on the surrounding context. To ensure accuracy and prevent "hallucinations," a heuristic is applied: the predicted word must start with the same letter as the original abbreviation.
The SBL-51abbr Dataset
To evaluate their approach, the researchers created SBL-51abbr, a human-curated gold-standard dataset. It consists of 51 randomly selected entries from the SBL, comprising 655 sentences and 2,041 abbreviations. Each entry is manually annotated for sentence boundaries, named entities, and the full grammatically correct expanded forms of all abbreviations.
Results and Impact
The impact of abbreviations on NLP performance is stark. In baseline tests on original text, the NER system achieved a macro-average F1 score of only 27.16. When abbreviations were manually expanded to their gold forms (the "ceiling"), this score surged to 59.31, highlighting the massive performance gap caused by unhandled abbreviations.
The proposed SloBERTa-based identification classifier reached an impressive F1 score of 95.97. More importantly, the automated expansion method successfully boosted the NER F1 score to 49.64. This represents a 22-point improvement over the baseline, effectively recovering a substantial portion of the performance lost to historical shorthand.
Conclusion and Future Directions
While the SloBERTa-based approach offers a significant leap forward, challenges remain. Slovenian's high morphological inflection makes it difficult for models to always predict the correct grammatical case or gender for an expanded word. Furthermore, the current heuristic expanded roughly 36% of abbreviations, leaving room for improvement in coverage. Future work will focus on integrating more sophisticated morphological modeling to ensure that expanded terms fit perfectly within the sentence's grammatical structure.
Summary updated