No article text was recorded for this version.
manual evaluation bottleneck
Version 1 of 154 papers56 conceptsfrozen 2026-10-04
Papers in this version
- Blog: FABLES: Evaluating faithfulness and content selection in book-length summarization
- Blog: WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation
- Blog: Meta-Evaluation of Sentence Simplification Metrics
- Blog: FFCI: A Framework for Interpretable Automatic Evaluation of Summarization
- Blog: Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
- Blog: Toward Human-Like Evaluation for Natural Language Generation with Error Analysis
- Blog: Overview of the TREC 2024 NeuCLIR Track
- Blog: BooookScore: A systematic exploration of book-length summarization in the era of LLMs
- Blog: Human Perception in Natural Language Generation
- Blog: Learning to Score System Summaries for Better Content Selection Evaluation.
- Blog: Model Criticism for Long-Form Text Generation
- Blog: Non-Expert Evaluation of Summarization Systems is Risky
- Blog: A Novel Evaluation Framework for Image2Text Generation
- Blog: LLM-based NLG Evaluation: Current Status and Challenges
- Blog: CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering
- Blog: BLEU Neighbors: A Reference-less Approach to Automatic Evaluation
- Blog: Automated Pyramid Summarization Evaluation
- Blog: Looking for a Few Good Metrics: Automatic Summarization Evaluation - How Many Samples Are Enough?
- Blog: Pragmatic Pertinence: A Learnable Confidence Metric to Assess the Subjective Quality of LM-Generated Text
- Blog: Automatic Evaluation of Text Coherence: Models and Representations
- Blog: ProxyQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models
- Blog: Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics
- Blog: Aligning Bilingual Literary Works: a Pilot Study
- Blog: From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
- Blog: Quality Estimation for Language Output Applications
- Blog: ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- Blog: Predicting the Usefulness of Amazon Reviews Using Off-The-Shelf Argumentation Mining
- Blog: ParaEval: Using Paraphrases to Evaluate Summaries Automatically
- Blog: LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs
- Blog: Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings
- Blog: Co-Eval: Augmenting LLM-based Evaluation with Machine Metrics
- Blog: Learning Summary Content Units with Topic Modeling
- Blog: An Automatic Method for Summary Evaluation Using Multiple Evaluation Results by a Manual Method
- Blog: Multilingual Summarization Evaluation without Human Models
- Blog: An Information-Theoretic Approach to Automatic Evaluation of Summaries
- Blog: FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation
- Blog: A LLM-based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation
- Blog: Investigating Memorization of Conspiracy Theories in Text Generation
- Blog: What is the Best Automated Metric for Text to Motion Generation?
- Blog: Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?
- Blog: Evaluating Information Content by Factoid Analysis: Human annotation and stability
- Blog: SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
- Blog: Automatic Evaluation of Information Ordering: Kendall's Tau
- Blog: Comparing Automatic Evaluation Measures for Image Description
- Blog: Finding a Balanced Degree of Automation for Summary Evaluation
- Blog: Describing a Knowledge Base
- Blog: Further Meta-Evaluation of Broad-Coverage Surface Realization
- Blog: Revisiting Summarization Evaluation for Scientific Articles
- Blog: PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment
- Blog: Tailoring and evaluating the Wikipedia for in-domain comparable corpora extraction
- Blog: SumPubMed: Summarization Dataset of PubMed Scientific Articles
- Blog: Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks
- Blog: Diversity-aware Evaluation for Paraphrase Patterns
- Blog: Looking hard: Eye tracking for detecting grammaticality of automatically compressed sentences
History & citation
Current · Permanent link
- Automated Proxies for Human Text Evaluation merged into this direction (77% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_manual_evaluation_bottleneck_43c7021f,
title = {manual evaluation bottleneck},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/manual-evaluation-bottleneck/rdr_43c7021f1fbc44febe294a4451b425b1},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}This exact version (version 1, frozen text):
@misc{lacuna_manual_evaluation_bottleneck_43c7021f_v1,
title = {manual evaluation bottleneck},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/manual-evaluation-bottleneck/rdr_43c7021f1fbc44febe294a4451b425b1/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}