TL;DR
This paper tackles the challenge of building a comprehensive dictionary for Singlish, a low-resource contact language, by leveraging ChatGPT. The authors demonstrate a novel, cost-effective method for generating dictionary entries, significantly accelerating the process compared to manual creation, and release the resulting open-source dictionary.
What Problem Does This Paper Address?
Modern Natural Language Processing (NLP) research heavily relies on vast amounts of data, which are readily available for resource-rich languages like English. However, this data-driven approach leaves many languages, particularly "low-resource" ones, behind. Singlish, a vibrant English-based creole spoken in Singapore, is a prime example. Despite its widespread use and unique linguistic features (syntactic, phonological, and lexical distinctions from Standard English), it lacks comprehensive, up-to-date dictionaries. Existing resources are often limited in scope, not open-source, or not actively maintained. This gap hinders further research and development in Singlish NLP applications.
Methods
The core of this project is the use of ChatGPT, a large language model (LLM), to generate draft dictionary entries. The process involved several key steps:
- Data Collection: A list of Singlish words and phrases was compiled from existing resources like the Dictionary of Singlish and Singapore English (DSSE), the Red Dot Baby Talk wordlist, and Wiktionary.
- LLM-based Generation: A specific prompt was designed to instruct ChatGPT to generate dictionary entries. This prompt requested a full description, part of speech (POS), pronunciation, and alternate spellings for a given Singlish word.
- Manual Annotation and Refinement: A principal annotator, a native Singlish speaker, manually reviewed and refined the ChatGPT-generated entries. This involved cross-validation with existing sources and native speaker intuition. Entries were tagged based on their accuracy and completeness.
- Pronunciation Normalization: Pronunciations, often inconsistently formatted by ChatGPT, were standardized into X-SAMPA notation.
- Inter-Annotator Agreement: To assess the reliability of the annotation process, a subset of entries was reviewed by multiple annotators, and their agreement scores were calculated.
Datasets
- Source Dictionaries for Word Collection:
- Dictionary of Singlish and Singapore English (DSSE)
- Red Dot Baby Talk wordlist
- Wiktionary
- Newly Introduced Benchmark: The paper introduces a new, open-source Singlish dictionary containing 1,783 entries, which is presented as the largest Singlish dictionary to date.
Key Insights and Ideas
- LLMs as Accelerators for Low-Resource Languages: ChatGPT proved to be a highly effective tool for rapidly generating draft entries for a low-resource language like Singlish, significantly reducing the manual effort required.
- The Importance of Human Oversight: While ChatGPT can generate a baseline, human annotation and refinement are crucial for ensuring accuracy, especially for nuanced linguistic features and examples.
- Granularity of Annotation Impacts Agreement: The initial, highly granular annotation tags led to lower inter-annotator agreement, highlighting the subjective nature of evaluating linguistic descriptions and the impact of annotator idiolects. Streamlining categories improved agreement.
- Singlish's Complexity: The study underscores the inherent complexity of Singlish, with its diverse origins, informal nature, and the significant role of tone, particularly in sentence-final particles.
Key Results
- Dictionary Size: The project successfully constructed a dictionary with 1,783 Singlish words and phrases.
- Annotation Accuracy: After manual review, approximately 46.2% of ChatGPT's descriptions were deemed completely satisfactory ('Yes').
- Efficiency: The LLM-based approach was found to be an efficient and low-cost method for creating and expanding a dictionary, with correcting entries being faster than writing them from scratch.
- Inter-Annotator Agreement: After streamlining annotation categories, the agreement score for "Useful Definition" reached 76.1%, with 74.3% of those agreeing that ChatGPT's outputs provided a correct definition.
Limitations and Future Work
- Limitations:
- Accuracy of LLM Outputs: ChatGPT's descriptions, while useful as a starting point, often contained inaccuracies, particularly in pronunciation and additional information beyond the core definition.
- Annotation Disagreement: Despite efforts to streamline, inter-annotator agreement remained a challenge, attributed to the granularity of tags and the inherent variation in Singlish usage and annotator interpretation.
- Incomplete Coverage: The current dictionary, while large, still has limitations and potential for further expansion.
- Future Work:
- Automated Augmentation: Developing automated processes to check and expand the dictionary by feeding new words found in descriptions back into ChatGPT.
- Corpus Integration: Semi-automatically feeding Singlish sentences from corpora (like the NUS SMS corpus) into ChatGPT to generate examples for dictionary entries.
- Prompt Engineering: Comparing the effectiveness of different prompts and exploring the use of multiple LLMs to generate more robust entries.
- Variation Exploration: Further investigating and accounting for the inherent variation in Singlish, potentially by recruiting annotators with more similar intuitions or by accepting slight variations in descriptions.
- WordNet Integration: Linking dictionary entries to WordNet senses for easier translation into other languages.
Summary updated