AcademicEval: Live Long-Context LLM Benchmark
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
Large Language Models (LLMs) have achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length and labor-intensive annotation, and the label leakage issue in LLM training also poses a pressing challenge. Therefore, we propose \textsc{AcademicEval}, a live benchmark for evaluating LLMs over long-context generation tasks. \textsc{AcademicEval} adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, \textit{i.e.}, \textsc{Title}, \textsc{Abstract}, \textsc{Introduction}, and \textsc{Related Work}, which cover a wide range of abstraction levels and require no manual labeling. Moreover, \textsc{AcademicEval} integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, \textsc{AcademicEval} features an efficient live evaluation, ensuring no label leakage. We conduct holistic experiments on \textsc{AcademicEval}, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, illustrating the challenge of our benchmark. We also provide insightful analysis for enhancing LLMs' long-context modeling capabilities.
BibTeX
@misc{zhang2025academiceval,
author = {Haozhen Zhang and Tao Feng and Pengrui Han and Jiaxuan You},
title = {{AcademicEval: Live Long-Context LLM Benchmark}},
year = {2025},
url = {https://openreview.net/forum?id=iRYExPKnxm},
howpublished = {OpenReview},
note = {Submitted to ICLR 2025},
}Loading related content...