This Word Mean What: Constructing a Singlish Dictionary with ChatGPT

Siew Yeng Chow, Chang-Uk Shin, Francis BondCOLING 2024

TL;DR

This paper tackles the challenge of building a comprehensive dictionary for Singlish, a low-resource contact language, by leveraging ChatGPT. The authors demonstrate a novel, cost-effective method for generating dictionary entries, significantly accelerating the process compared to manual creation, and release the resulting open-source dictionary.

What Problem Does This Paper Address?

Modern Natural Language Processing (NLP) research heavily relies on vast amounts of data, which are readily available for resource-rich languages like English. However, this data-driven approach leaves many languages, particularly "low-resource" ones, behind. Singlish, a vibrant English-based creole spoken in Singapore, is a prime example. Despite its widespread use and unique linguistic features (syntactic, phonological, and lexical distinctions from Standard English), it lacks comprehensive, up-to-date dictionaries. Existing resources are often limited in scope, not open-source, or not actively maintained. This gap hinders further research and development in Singlish NLP applications.

Methods

The core of this project is the use of ChatGPT, a large language model (LLM), to generate draft dictionary entries. The process involved several key steps:

  1. Data Collection: A list of Singlish words and phrases was compiled from existing resources like the Dictionary of Singlish and Singapore English (DSSE), the Red Dot Baby Talk wordlist, and Wiktionary.
  2. LLM-based Generation: A specific prompt was designed to instruct ChatGPT to generate dictionary entries. This prompt requested a full description, part of speech (POS), pronunciation, and alternate spellings for a given Singlish word.
  3. Manual Annotation and Refinement: A principal annotator, a native Singlish speaker, manually reviewed and refined the ChatGPT-generated entries. This involved cross-validation with existing sources and native speaker intuition. Entries were tagged based on their accuracy and completeness.
  4. Pronunciation Normalization: Pronunciations, often inconsistently formatted by ChatGPT, were standardized into X-SAMPA notation.
  5. Inter-Annotator Agreement: To assess the reliability of the annotation process, a subset of entries was reviewed by multiple annotators, and their agreement scores were calculated.

Datasets

  • Source Dictionaries for Word Collection:
    • Dictionary of Singlish and Singapore English (DSSE)
    • Red Dot Baby Talk wordlist
    • Wiktionary
  • Newly Introduced Benchmark: The paper introduces a new, open-source Singlish dictionary containing 1,783 entries, which is presented as the largest Singlish dictionary to date.

Key Insights and Ideas

  • LLMs as Accelerators for Low-Resource Languages: ChatGPT proved to be a highly effective tool for rapidly generating draft entries for a low-resource language like Singlish, significantly reducing the manual effort required.
  • The Importance of Human Oversight: While ChatGPT can generate a baseline, human annotation and refinement are crucial for ensuring accuracy, especially for nuanced linguistic features and examples.
  • Granularity of Annotation Impacts Agreement: The initial, highly granular annotation tags led to lower inter-annotator agreement, highlighting the subjective nature of evaluating linguistic descriptions and the impact of annotator idiolects. Streamlining categories improved agreement.
  • Singlish's Complexity: The study underscores the inherent complexity of Singlish, with its diverse origins, informal nature, and the significant role of tone, particularly in sentence-final particles.

Key Results

  • Dictionary Size: The project successfully constructed a dictionary with 1,783 Singlish words and phrases.
  • Annotation Accuracy: After manual review, approximately 46.2% of ChatGPT's descriptions were deemed completely satisfactory ('Yes').
  • Efficiency: The LLM-based approach was found to be an efficient and low-cost method for creating and expanding a dictionary, with correcting entries being faster than writing them from scratch.
  • Inter-Annotator Agreement: After streamlining annotation categories, the agreement score for "Useful Definition" reached 76.1%, with 74.3% of those agreeing that ChatGPT's outputs provided a correct definition.

Limitations and Future Work

  • Limitations:
    • Accuracy of LLM Outputs: ChatGPT's descriptions, while useful as a starting point, often contained inaccuracies, particularly in pronunciation and additional information beyond the core definition.
  • Annotation Disagreement: Despite efforts to streamline, inter-annotator agreement remained a challenge, attributed to the granularity of tags and the inherent variation in Singlish usage and annotator interpretation.
  • Incomplete Coverage: The current dictionary, while large, still has limitations and potential for further expansion.
  • Future Work:
    • Automated Augmentation: Developing automated processes to check and expand the dictionary by feeding new words found in descriptions back into ChatGPT.
  • Corpus Integration: Semi-automatically feeding Singlish sentences from corpora (like the NUS SMS corpus) into ChatGPT to generate examples for dictionary entries.
  • Prompt Engineering: Comparing the effectiveness of different prompts and exploring the use of multiple LLMs to generate more robust entries.
  • Variation Exploration: Further investigating and accounting for the inherent variation in Singlish, potentially by recruiting annotators with more similar intuitions or by accepting slight variations in descriptions.
  • WordNet Integration: Linking dictionary entries to WordNet senses for easier translation into other languages.

Summary updated

BibTeX
@inproceedings{chow2024this,
  author = {Siew Yeng Chow and Chang-Uk Shin and Francis Bond},
  title = {{This Word Mean What: Constructing a Singlish Dictionary with ChatGPT}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure