Towards a Danish Semantic Reasoning Benchmark - Compiled from Lexical-Semantic Resources for Assessing Selected Language Understanding Capabilities of Large Language Models

Bolette Pedersen, Nathalie Sørensen, Sussi Olsen, Sanni Nimb, Simon GrayCOLING 2024

TL;DR

This paper introduces the first Danish semantic reasoning benchmark, automatically compiled from existing lexical-semantic resources. It aims to provide a more robust evaluation of Large Language Models (LLMs) on nuanced Danish language understanding, going beyond simple translation tasks and addressing challenges like negation, contrast, and subtle word meanings.

What Problem Does This Paper Address?

As Large Language Models (LLMs) become increasingly sophisticated, there's a growing need for evaluation benchmarks that can truly test their understanding of language, especially for less-resourced languages like Danish. Existing benchmarks often rely on translation, which can introduce biases and fail to capture language-specific nuances. The authors highlight the challenge of creating comprehensive, high-quality semantic reasoning datasets for Danish from scratch, which is both time-consuming and prone to linguistic bias. They propose a solution by leveraging existing, human-curated lexical-semantic resources to automatically build a benchmark that probes deeper into LLMs' abilities in semantic inference, entailment, similarity, and word sense disambiguation.

Methods

The core of the paper's methodology lies in the semi-automatic compilation of datasets from four key Danish lexical-semantic resources:

  • DanNet (Danish WordNet): Used to infer ontological status and core meaning components of concepts, leveraging its hierarchical structure and predefined semantic relations (qualia roles).
  • Danish Thesaurus: Utilized to deduce semantic similarity and synonymy by analyzing word ordering and topical groupings.
  • Danish FrameNet Lexicon: Employed to extract semantic frames and their associated events, enabling the creation of entailment tests based on cause-and-effect relationships.
  • Central Word Register for Danish (COR.SEM): Used for word sense disambiguation tasks, particularly for a Word-in-Context (WiC) dataset.

The authors generated test instances using generic templates tailored to each resource and task. They also incorporated negative and contrastive examples, along with low-frequency vocabulary, to increase the difficulty and challenge LLMs beyond typical performance.

Datasets

The paper introduces a new benchmark for Danish, comprising 25 datasets across 6 different tasks, with a total of 3,800 test instances. These datasets are compiled from:

  • DanNet: Datasets for inference based on ontological types and qualia roles.
  • Danish FrameNet Lexicon: Datasets for entailment tests.
  • Danish Thesaurus: Datasets for synonymy, similarity, and relatedness tasks, including a "word intrusion" task where models must identify an outlier word.
  • Central Word Register for Danish (COR.SEM): A Danish Word-in-Context (WiC) dataset for word sense disambiguation.

This benchmark is made publicly available via GitHub.

Key Insights and Ideas

  • Leveraging Existing Resources: The paper demonstrates the feasibility of automatically compiling robust semantic evaluation datasets from existing, well-structured lexical resources, significantly reducing manual effort.
  • Beyond Prototypicality: The inclusion of negative, contrastive, and low-frequency examples is crucial for challenging LLMs and revealing their limitations in understanding nuanced language.
  • Danish-Specific Nuances: The benchmark aims to capture unique aspects of Danish vocabulary and semantics, avoiding biases often introduced by translating from other languages.
  • LLM Performance Variation: The evaluation of ChatGPT 3.5 and 4.0 reveals significant performance differences, with ChatGPT 4.0 generally outperforming its predecessor, but still struggling with certain complex tasks.
  • Resource Feedback Loop: The process of dataset compilation provided valuable feedback for improving the underlying lexical resources themselves, highlighting areas where peculiar meanings or abstract concepts can cause confusion.

Key Results

  • ChatGPT Performance: ChatGPT 4.0 achieved an average accuracy of 0.8 on the benchmark, while ChatGPT 3.5 turbo achieved an average of 0.6.
  • Task Performance: ChatGPT 4.0 showed high accuracy on synonymy (97%) and temporary/cause frames (100% and 95% respectively).
  • Challenging Tasks: The relatedness word intrusion task proved to be the most challenging, with models performing worse on less homogeneous core groups. The polysemous subset of the WiC task also remained challenging for both models.
  • Concrete vs. Abstract: Concrete ontological types generally led to better LLM performance than abstract ones.
  • TELLING Frames: These frames showed the lowest performance, suggesting that inferring the result of communication events is harder than for more concrete actions.

Limitations and Future Work

  • Dataset Size: The current version of the benchmark is still somewhat limited in size (3,800 instances).
  • Abstract Concepts: Abstract concepts and their placement in taxonomies proved more difficult to handle than concrete ones.
  • Peculiar Word Meanings: Old-fashioned or colloquial uses of words can lead to confusion and contra-intuitive utterances, suggesting a need to curate these out of lexical resources before dataset compilation.
  • FrameNet Challenges: Assigning frames to Danish verbs is complex due to polysemy and the tendency for verbs to mutate into phrasal verbs, making context crucial for frame identification.
  • Future Work: The authors plan to scale up the datasets, increase the difficulty of tasks (e.g., testing "tone of voice" for synonyms, metaphorical expressions), and expand to other reasoning tasks. They also aim to test smaller language models and investigate the correlation between benchmark performance and performance on regular downstream tasks. Further refinement of the lexical resources based on the feedback from dataset compilation is also planned.

Summary updated

BibTeX
@inproceedings{pedersen2024towards,
  author = {Bolette Pedersen and Nathalie Sørensen and Sussi Olsen and Sanni Nimb and Simon Gray},
  title = {{Towards a Danish Semantic Reasoning Benchmark - Compiled from Lexical-Semantic Resources for Assessing Selected Language Understanding Capabilities of Large Language Models}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure