TL;DR
This paper surveys how human evaluation is conducted in creative Natural Language Generation (NLG) research, bridging the gap between Computational Creativity (CC) and Natural Language Processing (NLP). It reveals a lack of standardized practices and clear motivations for evaluation parameters, urging for more rigorous and transparent methodologies to advance the field.
What Problem Does This Paper Address?
Evaluating creative output is inherently challenging, especially in Natural Language Generation (NLG). While traditional NLG focuses on factual accuracy and fluency, "creative NLG" aims for aesthetic qualities like novelty, emotion, and artistic value. This paper addresses the problem of how this creative output is evaluated by humans. Existing surveys on NLG evaluation have largely focused on non-creative tasks and haven't distinguished between the approaches of NLP and Computational Creativity (CC) fields. The authors aim to bridge this gap by surveying recent papers from both ICCC (a key CC venue) and INLG (a key NLP venue) to understand their human evaluation methodologies, identify commonalities and differences, and suggest improvements.
Methods
The authors conducted a literature review of papers presented at ICCC 2020 and INLG 2020 that involved human evaluation of creative NLG systems. They specifically selected papers dealing with NLG tasks where the output could exhibit creativity, excluding purely factual data-to-text generation. For each selected paper, they analyzed:
- The specific NLG task.
- The parameters being evaluated (e.g., meaning, novelty, emotional value).
- The motivation behind choosing these parameters.
- The methodological approach to human evaluation (e.g., scaled surveys, preference rankings).
- The evaluation setup, including the number of evaluators and samples, and whether evaluators were experts or laymen.
- How the evaluation results were discussed and used for future directions.
Datasets
The paper does not introduce new datasets. Instead, it surveys existing research that utilized human evaluation on various creative NLG tasks. The specific datasets used within those surveyed papers are not detailed in this survey paper.
Key Insights and Ideas
- Lack of Consensus: There's no established consensus on how to evaluate creative NLG, despite efforts to create guidelines.
- Dominant but Varied Methods: Scaled surveys (often 5-point) are the most common evaluation method, but the specific questions and parameters vary widely.
- Commonly Evaluated Parameters: Meaning, syntactic correctness, novelty, relevance, and emotional value are frequently assessed, though often with different terminology.
- Weak Motivation for Parameters: Most papers fail to provide clear reasoning for why specific parameters were chosen for evaluation, making the choices seem arbitrary.
- Ad Hoc Evaluation Design: Evaluation setups are often created from scratch for each paper, with little reference to existing research or pre-testing, leading to potential issues with question interpretation and validity.
- Limited Use of Experts: While experts are sometimes used (especially for specialized tasks), many papers rely on non-expert annotators, raising questions about their suitability for nuanced creative evaluation.
- Underutilization of Results: Evaluation results are often used merely to show one system is better than another, rather than to derive concrete insights for improving the system or understanding its limitations.
- Interdisciplinary Differences: While both fields use human evaluation, the CC field sometimes presents more exploratory research, while NLP is often more state-of-the-art driven.
Key Results
- Survey Scope: The survey covered 12 papers from ICCC 2020 and 11 papers from INLG 2020.
- Evaluation Methods:
- Scaled Surveys: The most common, with 5-point scales used in papers 4, 7, 9 (eval 1), 11, 12, 14, 15, 18, and 21 (eval 1). Other scales (3, 4, 7 points) and continuous sliders were also used.
- Preference-Based: Ranking models or picking the best output was also frequent (e.g., papers 1, 10, 13, 23).
- User Studies: Qualitative (paper 2) and quantitative (paper 17) user studies were less common.
- Evaluated Parameters: Meaning (papers 4, 5, 10, 14, 19), syntactic correctness/fluency (papers 9, 13, 14, 11, 12, 8, 18), novelty/originality/surprise (papers 5, 9, 7), and relevance/relatedness (papers 18, 20, 6, 8, 16) were among the most frequent. Emotional value (papers 9, 10, 11, 7, 6, 8, 16) was also prominent.
- Sample Sizes and Annotators: Sample sizes varied greatly (from 2 to 250 artifacts). Annotator numbers were often small (typically 3-5), and demographic information was rarely reported.
Limitations and Future Work
- Limitations:
- Lack of Motivation: Many papers don't justify their choice of evaluation parameters.
- Unclear Questions: Actual evaluation questions are often not revealed, making it hard to assess their validity.
- Untested Methods: Evaluation setups are rarely pre-tested, leading to potential misinterpretations by annotators.
- Bias: Lack of demographic data for annotators and discussion of potential biases (e.g., priming, selection bias) is a significant limitation.
- Small Sample Sizes: The number of annotators and samples is often too small for robust conclusions.
- Limited Use of Results: Evaluation results are not deeply analyzed to guide future research or identify specific system shortcomings.
- Future Work/Recommendations:
- Define Goals Clearly: Precisely state the system's goals to motivate evaluation parameters.
- Be Concrete: Use specific, unambiguous questions to reduce subjective interpretation.
- Test Evaluation Setups: Conduct pilot studies to refine questionnaires and methods.
- Use Multiple Evaluation Setups: Employ diverse methods (e.g., preference, scaled, qualitative) for a more comprehensive understanding.
- Report Everything Clearly: Document the entire evaluation process, including instructions, wording, and potential biases.
- Analyze Results Profoundly: Go beyond basic statistics to conduct deeper error analysis and identify actionable insights.
- Learn from Other Fields: Incorporate established methodologies from human-computer interaction and survey research.
Summary updated