TL;DR
This paper introduces an "omni-font" system for automatically converting Gurmukhi (Punjabi) text from various legacy fonts into Unicode, and then transliterating it into Shahmukhi. This addresses the significant challenge of digitizing and preserving Punjabi content trapped in older, non-standardized font encodings, achieving high accuracy in both font detection and transliteration.
What Problem Does This Paper Address?
Punjabi, written in the Gurmukhi script, has a vast amount of historical and contemporary text locked away in legacy ASCII-based fonts. Unlike Unicode, these older fonts lack a standardized mapping of characters, meaning the same Gurmukhi character can be represented by different internal codes in different fonts. This makes it incredibly difficult to process, search, and digitize this content. While some font converters exist, they often require users to manually identify the source font, which is not always feasible. This paper tackles the problem of automatically identifying the font and then accurately converting the text to a standard format (Unicode) and subsequently to the Shahmukhi script.
Methods
The system operates in two main stages:
- Omni-Font Detection and Conversion to Unicode:
- Font Detection: This stage uses a character-level trigram language model. Instead of treating each of the 225+ Gurmukhi fonts as a separate class, the system groups them into 41 "keyboard map families." The core idea is that fonts belonging to the same keyboard map family share similar internal character mappings. The system extracts character trigrams from the input text and calculates their probability within each of the 41 keyboard map models. The keyboard map with the highest probability is identified.
- Unicode Conversion: Once the keyboard map is identified, the system uses a handcrafted mapping table for that specific map to convert the input text into Unicode characters. Special attention is paid to handling complex vowel and consonant combinations to ensure accurate representation in Unicode.
- Gurmukhi to Shahmukhi Transliteration:
- Pre-Processing: The Unicode text undergoes spell-checking and normalization, aligning it with Shahmukhi spellings and pronunciation.
- Transliteration Engine: This stage employs a multi-pronged approach:
- Dictionary Lookup: For frequently occurring words and proper nouns, a direct lookup in a created Gurmukhi-Shahmukhi dictionary is performed.
- Gurmukhi Stemmer: For words not found in the dictionary, a stemming process is applied to remove common suffixes, and the root word is then searched.
- Rule-based Transliteration: If dictionary lookup and stemming fail, a direct grapheme-based mapping is used, where Gurmukhi characters are mapped to similar-sounding Shahmukhi characters. Special rules handle vowel combinations and consonant clusters.
- Word Disambiguation: For characters with multiple possible Shahmukhi mappings, a Hidden Markov Model (HMM) is used to select the most probable interpretation based on context and unigram frequencies.
Datasets
- Training Data: An 11 million word corpus was used for training the trigram language model for font detection.
- Evaluation Data:
- For font detection and Unicode conversion, a test set of 194 words (962 characters) was used, converted into 35 popular Gurmukhi fonts.
- For transliteration accuracy, over 100 pages of text compiled from newspapers, books, and poetry were used.
Key Insights and Ideas
- Leveraging Keyboard Maps: The insight that many legacy Gurmukhi fonts can be grouped into a smaller number of "keyboard map families" significantly reduces the complexity of the font detection problem from hundreds of classes to just 41.
- Trigram Language Models for Font Detection: Applying character-level trigram models, typically used in NLP for language modeling, to the problem of font identification is an innovative approach. The probability of trigram sequences within a specific keyboard mapping effectively acts as a fingerprint for that mapping.
- Multi-Stage Transliteration: The combination of dictionary lookup, stemming, rule-based mapping, and HMM-based disambiguation creates a robust transliteration engine that can handle a wide variety of linguistic phenomena.
- Handling Unicode Nuances: The paper highlights the specific challenges of mapping complex Gurmukhi vowel and consonant structures to Unicode, particularly the absence of explicit code points for half characters, which rely on rendering systems.
Key Results
- Font Detection and Unicode Conversion Accuracy: The system achieved 99.73% accuracy at the word level for font detection and conversion to Unicode, with TOP1 font detection. While 3 out of 35 fonts were not perfectly recognized, the confusion was minor and did not significantly impact subsequent stages.
- Gurmukhi to Shahmukhi Transliteration Accuracy: The overall transliteration accuracy is reported as 98.6% at the word level.
Limitations and Future Work
- Limitations:
- The system experienced confusion with fonts that had very similar character mapping tables (differing by only 2-3 code maps).
- The transliteration accuracy can be affected by typical spellings in the input words.
- Future Work:
- Increasing the size of the training corpus for the trigram model could further improve font detection accuracy.
- Expanding the Shahmukhi word frequency list and covering more ambiguous word senses would enhance transliteration accuracy.
- The authors suggest that the system's accuracy can be further increased by increasing the training corpus size and providing more data covering maximum senses of ambiguous words in the target script.
Summary updated