No article text was recorded for this version.
execution-based evaluation
Version 2 of 2253 papers308 conceptsfrozen 2026-10-04
Papers in this version
- Blog: InstructCoder: Empowering Language Models for Code Editing
- Blog: Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'
- Blog: CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
- Blog: NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands
- Blog: ExecRepoBench: Multi-level Executable Code Completion Evaluation
- Blog: Deep Learning for Source Code Modeling and Generation: Models, Applications, and Challenges
- Blog: GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities
- Blog: DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
- Blog: DebateCoder: Towards Collective Intelligence of LLMs via Test Case Driven LLM Debate for Code Generation
- Blog: Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities
- Blog: XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval
- Blog: Function-to-Style Guidance of LLMs for Code Translation
- Blog: AVATAR: A Parallel Corpus for Java-Python Program Translation
- Blog: Large Language Model Critics for Execution-Free Evaluation of Code Changes
- Blog: GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models
- Blog: SPoC: Search-based Pseudocode to Code
- Blog: MPIrigen: MPI Code Generation through Domain-Specific Language Models
- Blog: DocGenBench: A Benchmark for Evaluating Code Documentation Agents
- Blog: R2E: Turning any Github Repository into a Programming Agent Environment
- Blog: BackportBench: A Multilingual Benchmark for Automated Backporting of Patches
- Blog: QiMeng-MuPa: Mutual-Supervised Learning for Sequential-to-Parallel Code Translation
- Blog: A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability
- Blog: CodeScore: Evaluating Code Generation by Learning Code Execution
- Blog: Precise and Interpretable Editing of Code Knowledge in Large Language Models
- Blog: Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation
- Blog: DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
- Blog: Mutual-Supervised Learning for Sequential-to-Parallel Code Translation
- Blog: NoviCode: Generating Programs from Natural Language Utterances by Novices
- Blog: Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking
- Blog: GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
- Blog: CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
- Blog: Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
- Blog: Execution-based Evaluation for Data Science Code Generation Models
- Blog: FixEval: Execution-based Evaluation of Program Fixes for Programming Problems
- Blog: Detecting Functional Memorization in Code Language Models
- Blog: InstructCoder: Instruction Tuning Large Language Models for Code Editing
- Blog: Deep Learning Based Code Generation Methods: Literature Review
- Blog: FixEval: Execution-based Evaluation of Program Fixes for Competitive Programming Problems
- Blog: Generative Software Engineering
- Blog: CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation
- Blog: VERSE: Verification-based Self-Play for Code Instructions
- Blog: ExecRepoBench: Multi-level Executable Code Completion Evaluation
- Blog: DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
- Blog: Detecting Functional Memorization in Code Language Models
- Blog: A Survey on Semantic Parsing
- Blog: Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey
- Blog: FaithfulPersona: Balancing Faithfulness and Personalization in Code Explanations through Self-Critique
- Blog: Instruction Induction: From Few Examples to Natural Language Task Descriptions
- Blog: DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
- Blog: Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain
- Blog: OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
- Blog: Neural Rankers for Code Generation via Inter-Cluster Modeling
- Blog: AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation
- Blog: What If: Generating Code to Answer Simulation Questions in Chemistry Texts
- Blog: CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs
- Blog: TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models
- Blog: NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
- Blog: OSS-Bench: Benchmark Generator for Coding LLMs
- Blog: AL-Bench: A Benchmark for Automatic Logging
- Blog: PICASO: Enhancing API Recommendations with Relevant Stack Overflow Posts
Showing 60 of 253 papers.
History & citation
Current · Permanent link
- cross-paradigm type inference split from this direction (50% overlap)
- functional correctness evaluation split from this direction (56% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_execution_based_evaluation_41027319,
title = {execution-based evaluation},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/execution-based-evaluation/rdr_41027319cc474b39858abf8bb8a46651},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}This exact version (version 2, frozen text):
@misc{lacuna_execution_based_evaluation_41027319_v2,
title = {execution-based evaluation},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/execution-based-evaluation/rdr_41027319cc474b39858abf8bb8a46651/v/2},
note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}