Abstract
Large Language Model (LLM) performance is highly sensitive to prompt engineering, yet current benchmarking lacks a standardized method for reporting prompt complexity. This paper introduces TELeR, a taxonomy designed to categorize prompts across four dimensions: Turn, Expression, Level of Details, and Role. By providing a common language for prompt description, TELeR enables "apple-to-apple" comparisons in LLM research, particularly for ill-defined complex tasks.
Introduction
In the current LLM research landscape, benchmarking is often inconsistent. A model's success or failure on a specific task can vary wildly depending on the prompt's detail and structure. This lack of standardization makes it difficult to determine whether performance gains are due to model improvements or superior prompt engineering. This issue is particularly acute for complex tasks—such as scientific meta-reviewing or narrative braiding—which involve multiple sub-steps and subjective interpretation. Without a standardized reporting framework, a performance metric like "90% accuracy" lacks context regarding the level of guidance provided to the model.
The TELeR Taxonomy
To address this lack of standardization, the authors propose the TELeR taxonomy. It classifies prompts based on four key dimensions: Turn, Expression, Level of Details, and Role. By using this notation—for example, <Single, Instruction, L3, Undefined>—researchers can precisely communicate the "strength" of the prompt used in their experiments.
The "Level of Details" dimension is the core of the taxonomy, providing a granular scale for prompt complexity:
- Level 0 (L0): Just data, no instructions.
- Level 1 (L1): A simple one-sentence directive expressing the high-level goal.
- Level 2-3 (L2-L3): Increasing detail with paragraph-style or bulleted sub-tasks.
- Level 4 (L4): Includes evaluation criteria or few-shot examples.
- Level 5 (L5): Includes external information gathered via retrieval-based techniques (RAG).
- Level 6 (L6): Explicitly requires the model to provide a justification or explanation for its output.
Demonstrating the Framework
The authors demonstrate the taxonomy using two complex use-case scenarios: Meta-Review Generation (summarizing multiple peer-review reports) and Narrative Braiding (interweaving independent storylines). By providing concrete prompt examples for these tasks at every level of the TELeR hierarchy, the paper shows how the taxonomy applies to real-world research across different domains, from scientific summarization to creative writing.
Key Insights and Ideas
The primary insight of TELeR is that standardization is essential for reproducibility. By moving prompt engineering from an art to a descriptive science, researchers can better attribute model performance. The "Detail Spectrum" (L0-L6) highlights that "zero-shot" prompting is not a monolith; there is a significant difference between a bare-bones L1 prompt and a sophisticated L4 prompt. For complex tasks, higher levels of detail (L5-L6) are often necessary to achieve human-like performance, requiring planning, semantic understanding, and often external data.
Limitations and Future Work
While TELeR provides a robust framework for complex tasks, it may be excessive for simple NLP tasks like sentiment analysis or basic named entity recognition. The authors also note that while the levels are defined, some subjectivity remains in distinguishing between adjacent levels (e.g., L2 vs. L3). Future iterations of the taxonomy could include additional dimensions such as temperature settings or specific decoding strategies. The ultimate goal is for the community to adopt TELeR as a standard reporting requirement in major NLP conferences to ensure transparency in how LLMs are evaluated.
Summary updated