Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses

Subin An, Yugyeong Ji, Junyoung Kim, Heejin Kook, Yang Lu, Josh SeltzerEMNLP 2025

TL;DR

This paper introduces a novel, transparent, and reference-free framework for automatically evaluating the quality of open-ended user survey responses. By focusing on human-written text characteristics like effort, relevance, and completeness, it outperforms existing methods and offers practical applications for improving survey data analysis and quality control.

What Problem Does This Paper Address?

Marketing research heavily relies on open-ended survey responses for rich, nuanced insights. However, the quality of these responses varies significantly, with low-quality or nonsensical answers burdening researchers with manual filtering and potentially leading to flawed conclusions. Existing automated evaluation methods, often designed for LLM-generated text, are ill-suited for the unique, often informal and less structured nature of human-written responses. Furthermore, traditional reference-based metrics (like BLEU or ROUGE) are impractical for open-ended survey data where ground truth references are scarce. This paper addresses the need for an automated evaluation system that can accurately and transparently assess human-written survey responses without requiring predefined references.

Methods

The proposed framework employs a two-stage approach:

  1. Gibberish Filtering: This initial stage removes nonsensical or malformed responses. It utilizes language-specific pipelines for English and Korean, incorporating statistical modeling and linguistic heuristics. This includes detecting unusual consonant/vowel sequences, checking the ratio of valid English words, and handling abbreviations/slang. For Korean, it involves analyzing syllabic completeness and Jamo diversity, along with morphological analysis.
  2. Dimension-wise Evaluation: This stage uses Large Language Models (LLMs) as judges to assess responses across three empirically derived dimensions:
  3. Effort: Evaluates the cognitive effort, informativeness, and specificity of the response. It goes beyond simple length to consider the depth of thought and detail provided.
  4. Relevance: Assesses how well the response aligns with both the topic and the intent of the question. This is crucial for understanding the implicit meaning and purpose behind the user's answer.
  5. Completeness: Measures the extent to which the response fulfills the core informational requirements of the question, ensuring all essential elements are addressed.

Each dimension is evaluated independently using a prompt-based LLM inference, with scoring rubrics and justifications to ensure interpretability and consistency.

Datasets

The study utilized human-written open-ended survey responses collected from real-world datasets provided by Nexxt Intelligence. These datasets were available in both English and Korean.

  • English Dataset: 275 responses (181 for development, 94 for testing).
  • Korean Dataset: 222 responses (122 for development, 100 for testing).

Responses were sampled to ensure diversity in question types, lengths, and quality levels. For Korean data annotation, question-response pairs were translated into English to maintain consistency with the annotation process. The paper does not introduce new benchmarks but uses existing datasets for evaluation.

Key Insights and Ideas

  • Human responses are fundamentally different from LLM outputs: The paper highlights that human responses are often shorter, more casual, and subjective, necessitating evaluation criteria tailored to these characteristics rather than those optimized for LLM-generated text.
  • Effort is more than just length: Simply measuring response length is insufficient for assessing effort. The framework's success in capturing informativeness and specificity shows that content-aware evaluation is key.
  • Intent alignment is critical for relevance: Traditional similarity measures often fail to capture the true relevance of a response. The paper demonstrates that considering both topic and intent alignment, as done by the LLM-based approach, leads to more human-aligned relevance assessments.
  • Completeness needs to go beyond surface-level features: Concise yet complete human responses can be undervalued by metrics relying on length or keyword count. The proposed method's success in completeness evaluation suggests it captures implicit cues and contextual fulfillment.
  • LLMs can be effective judges for human text: The framework leverages LLMs' language understanding to provide nuanced, multi-dimensional evaluations without needing references, overcoming limitations of traditional metrics.
  • Transparency through justifications: The LLM's ability to provide brief justifications for each score enhances interpretability, making the evaluation process more transparent for researchers.

Key Results

  • Superior performance across dimensions: The proposed framework consistently outperformed baseline methods (length-based, SBERT, DeepEval) in evaluating effort, relevance, and completeness on both English and Korean datasets, showing strong Spearman and Kendall correlations with human expert ratings. For instance, in English, the framework achieved an 0.8198 Spearman correlation for effort, compared to 0.7889 for length-based.
  • High accuracy in gibberish detection: The gibberish detection module achieved an F1 score, precision, recall, and accuracy of 0.98 for both English and Korean datasets, significantly outperforming BERT-based models and LLMs alone.
  • Practical utility demonstrated: The aggregated dimension scores were successfully applied to two downstream tasks:
  • Response Quality Prediction: Achieved high correlations with expert ratings (e.g., AUC of 0.97 for English and 0.83 for Korean in acceptance classification).
  • Response Rejection: The framework's scores can be operationalized with thresholds for automated quality control.
  • Cross-lingual applicability: The framework's strong performance on both English and Korean datasets demonstrates its robustness across languages.

Limitations and Future Work

Limitations: * LLM subjectivity: The scoring process still relies on the subjective inference of LLMs, and biases learned during pretraining can be difficult to fully eliminate. * Lack of explicit rationale: While LLMs provide justifications, they don't reveal the underlying token-level evidence, limiting rigorous auditing of the evaluation outcomes. * Need for broader theoretical grounding: The three dimensions (effort, relevance, completeness) were introduced based on practical considerations; further investigation is needed to establish them as general, theoretically grounded criteria. * Data privacy: The specific dataset used in the study cannot be made publicly available due to confidentiality, limiting full transparency and reproducibility.

Future Work: * Agentic and Real-Time Integration: Powering multi-turn, agentic survey platforms that evaluate responses in real-time, flag shortcomings, and generate targeted follow-up questions to improve data quality on the spot. * Safeguarding Data Quality: Integrating detection of low-effort or insincere responses directly into survey workflows to preserve validity and reduce downstream noise. * Cross-Lingual and Cross-Cultural Robustness: Extending the framework beyond English and Korean to ensure generalization across diverse populations and identify/mitigate biases in multilingual contexts. * Pathway to Quantitative Analysis: Leveraging the interpretability of dimension-wise scores for quantitative utilization, embedding them into broader analytic pipelines for higher-order insights.

Summary updated

BibTeX
@inproceedings{an2025transparent,
  author = {Subin An and Yugyeong Ji and Junyoung Kim and Heejin Kook and Yang Lu and Josh Seltzer},
  title = {{Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure