Overview of ELOQUENT 2024 - Shared Tasks for Evaluating Generative Language Model Quality
Lists
Only you see this
Abstract
ELOQUENT is a set of shared tasks for evaluating the quality and usefulness of generative language models. ELOQUENT aims to apply high-level quality criteria, grounded in experiences from deploying models in real-life tasks, and to formulate tests for those criteria, preferably implemented to require minimal human assessment effort and in a multilingual setting. The tasks for the first year of ELOQUENT were (1) Topical quiz, in which language models are probed for topical competence; (2) HalluciGen, in which we assessed the ability of models to generate and detect hallucinations; (3) Robustness, in which we assessed the robustness and consistency of a model output given variation in the input prompts; and (4) Voight-Kampff, run in partnership with the PAN lab, with the aim of discovering whether it is possible to automatically distinguish human-generated text from machine-generated text. This first year of experimentation has shown—as expected—that using self-assessment with models judging models is feasible, but not entirely straight-forward, and that a a judicious comparison with human assessment and application context is necessary to be able to trust self-assessed quality judgments.
BibTeX
@inproceedings{karlgren2024overview,
author = {Jussi Karlgren and Luise Dürlich and Evangelia Gogoulou and Liane Guillou and Joakim Nivre and Magnus Sahlgren and Aarne Talman and Shorouq Zahra},
title = {{Overview of ELOQUENT 2024 - Shared Tasks for Evaluating Generative Language Model Quality}},
booktitle = {CLEF},
year = {2024},
doi = {10.1007/978-3-031-71908-0\_3},
}Loading related content...