On public blockchains, transactions sit in a public waiting room — the mempool — before they are executed, and anyone can read them there. Automated bots exploit this transparency by reordering transactions for profit: they front-run a pending trade, or "sandwich" it between a buy and a sell to capture the price move the victim's own trade creates Torres et al.. A growing family of LLM-based smart-contract auditors claims to detect such vulnerabilities, and their reported numbers look strong. But there is a catch: the hacks they are evaluated on — famous exploits, archived audit contests, curated CVE lists — are all over the internet, and therefore almost certainly inside every model's training data. Nobody currently knows whether these systems reason about transaction ordering or merely recite memorized post-mortems.
This proposal builds the instrument that answers the question, and uses it to test a specific remedy. I will construct a continuously refreshed, timestamp-stamped evaluation stream of DeFi contracts, audit contest submissions, and mined mempool attack instances, restricted — per model — to items created after that model's training cutoff, applying the causal firewall already proven in coding evaluation by LiveCodeBench and the history-pruning protocol of SWE-STEP Shastry et al.. On this stream I will measure how much of reported auditing performance is contamination, and whether grounding reasoning in executable simulation — prompting the model to write and run a script that computes sandwich-attack profitability — recovers what natural-language reasoning and definition-stuffing do not. This is a measurement study plus a method comparison, not a defense proposal; private transaction routing moves the problem rather than solving it Weintraub et al..
Core Claim
The claim has three parts, each independently falsifiable. First, LLM front-running detection scores on static datasets are substantially inflated by contamination: under temporal isolation, the same pipelines will drop sharply, in the same way that apparent causal reasoning collapsed when Chi et al. moved evaluation to fresh post-cutoff data. Second, a large share of the remaining failure is a domain-knowledge problem of a particular kind — models know what a sandwich attack is, but cannot track state across inter-contract execution paths, the exact blind spot that Zhang et al. identified when 222 of 383 sampled missed attacks involved inter-contract calls that no conventional tool could follow. Third, Program-of-Thought prompting — the model writes executable code instead of prose reasoning, as in TheoremQA — closes much of that residual gap on ordering questions, because attack profitability is a computable function of pool reserves, trade size, and gas cost, not a fact to be recalled.
Research Program
The artifact is a benchmark, provisionally called LiveMEV, with two task families and a fixed intervention grid.
The stream is assembled with the mining machinery Zhang et al. already validated: their taint-based attack miner achieved 98.69% precision on Ethereum blocks and found 24.42× more attacks than a baseline method Zhang et al.. I will run an analogous miner over blocks mined after the evaluation date, and pair those attack instances with (a) newly published audit-contest contracts, stamped with contest close dates, and (b) confirmed exploit post-mortems, stamped with disclosure dates. Every item carries a timestamp, and a model is only ever evaluated on items post-dating its cutoff — the LiveCodeBench recipe transplanted from competitive programming to DeFi. Task family A is vulnerability identification: given a contract together with its full inter-contract call context (deliberately unlike single-contract auditors), decide whether a specified function admits ordering-based extraction, with ground truth from contest judge reports and confirmed exploits. Task family B is profitability prediction: given pool reserves, a pending victim trade, and gas prices, decide whether a sandwich attack is profitable and what slippage would make the victim safe. Ground truth for family B is exact, computed by simulation from the sandwich-game analysis of Heimbach et al., who showed that attacks are eliminated whenever the victim's slippage tolerance $s$ sits between the volatility buffer $s_r$ and the unprofitability threshold $s_a$:
$$\text{attack eliminated} \iff s_r \le s < s_a,$$
where $s_a$ is the slippage at which the bot's captured spread no longer covers the gas cost of its two transactions. This inequality is the key object the reasoning conditions must recover: it depends on pool reserves, the victim's trade size, and gas prices, all of which vary item to item.
The intervention grid crosses three reasoning channels with two knowledge sources. The reasoning channels are: direct zero-shot answering; natural-language chain-of-thought, in which the model writes out its intermediate steps before deciding — the mechanism by which, as Huang et al. showed, deliberation helped GPT-3.5 find game-theoretic equilibria, lifting its Diner's Dilemma score by 78.5 points; and Program-of-Thought, in which the model emits a Python script that is actually executed against a pool simulator, per the TheoremQA finding that offloading computation to an interpreter took GPT-4 to 51.0% where open models stayed below 15% Chen et al.. The knowledge sources are: a synthesizer built only from literature predating the model's cutoff (a temporally honest version of the AiRacleX knowledge-synthesis stage AiRacleX), and an adaptive synthesizer whose retrieval is truncated to publications predating each test item, borrowing the temporal search truncation of ARAC. This contrast directly probes the tension AiRacleX leaves open: static academic synthesis anchors the auditor to historical attack templates, while the MEV survey documents that attack patterns constantly mutate into multi-address and cross-domain forms MEV survey, and Zhu et al. show models degenerate precisely when real-world mechanisms move past their training distribution.
Evaluation models will include a frontier closed-weight model and an open-weight model at the LLaMA-3.1-405B / Command R+ scale, since Zhu et al. found open models more temporally stable than GPT-4o and Claude 3.5 Sonnet — a difference worth confirming on a new domain. A secondary measurement tracks reasoning length and wall-clock time per decision against a block-inclusion deadline, because Si et al. showed that deliberating models can be trapped in unbounded reasoning loops, and a mempool decision that arrives after the block is worthless.
Baselines
The naive reference point is direct zero-shot prompting of the frontier model with no reasoning chain and no retrieved knowledge, run on the live stream exactly as on any other condition. This is the "do nothing" number: whatever it scores on post-cutoff items is the true unscaffolded capability, and every structured method must clear it by a margin beyond seed variance to justify its existence. On static data, the analogous gap is already visible — LLM-SmartAudit reports 84% accuracy for generic prompts versus 98% for its structured targeted-analysis mode on common vulnerabilities — but that comparison is itself contamination-suspect, which is precisely what the stream resolves.
The strongest existing competitor is the AiRacleX pipeline, which synthesizes domain knowledge from academic papers, converts it into structured chain-of-thought prompts, and audits contracts with them; on its static DeFiHack-plus-Code4rena set it reports recall of 0.667 and F1 of 0.426, versus 0.259 recall and 0.283 F1 for GPTScan — a 2.58× recall improvement AiRacleX. It is the strongest because it is the only prior system that combines LLM reasoning, mined domain knowledge, and structured prompting on real DeFi vulnerabilities. I will re-implement its three-stage pipeline faithfully and run it on the live stream with one integrity constraint: its knowledge synthesizer may only see pre-cutoff literature. Both it and my conditions use the same model backbones, the same per-item token and retrieval budget, and the same tuning protocol — a fixed configuration grid tuned on a small pre-cutoff development split that never touches the stream. Beating it means higher recall on post-cutoff items at matched false-positive rate, plus a smaller pre-cutoff-to-post-cutoff performance drop. I must be explicit that no reported result found on Lacuna for any LLM-based front-running detector under temporal isolation exists; AiRacleX's 0.667 recall is a static-data number, and part of this study's point is that it should not be expected to transfer.
The conventional-tool floor comes from Zhang et al.'s benchmark of 513 real-world front-running attacks across 235 contracts, where the best of seven static and dynamic vulnerability detection tools achieved recall of only 6.04% Zhang et al.. I will re-run the top tools from that study on the stream's contract items under their original configurations; they anchor the bottom of the chart and also supply the labeling machinery for the stream itself.
The key ablation holds everything fixed except the reasoning channel: identical model, identical retrieved knowledge corpus, identical token budget, with only the output mode varying among zero-shot, natural-language chain-of-thought, and executable Program-of-Thought. A second ablation holds the reasoning channel fixed and varies only the knowledge source between pre-cutoff-only and item-adaptive synthesis. These two ablations isolate, respectively, the executable-simulation ingredient and the temporal-adaptivity ingredient — the two things this proposal claims matter.
Why not just…?
Why not just evaluate on the static benchmarks that already exist? Because the static sets are the confound, not the control. Contamination is documented at web scale Ravaut et al., and Chi et al. showed that performance which looks like reasoning on static data collapses on fresh post-cutoff distributions, revealing memorized patterns. Concretely, LLM-SmartAudit's celebrated 98% is on a curated benchmark of well-known vulnerabilities, while its accuracy on the fresher real-world Code4rena corpus was 47.6% LLM-SmartAudit — a 50-point gap that contamination is the natural suspect for. The decisive experiment inside this proposal is the matched-pairs comparison: identical attack patterns, one instance pre-cutoff and one post-cutoff. If the performance gap is near zero, the contamination claim is falsified and the static-benchmark skeptics win; I would report that outcome prominently, because a clean negative would itself settle an open question.
Why not just prompt a frontier model zero-shot and trust scale? The evidence says scale alone does not fix domain deficits: parameter scaling fails to compensate for missing niche distributions Chen et al., and on TheoremQA every open model scored below 15%, barely above random guess, while GPT-4 needed programmatic offloading to reach 51.0% Chen et al.. GAMA-Bench cuts the other way and I take it seriously: chain-of-thought there was the difference between missing and finding a Nash equilibrium Huang et al., so plain natural-language reasoning may already be enough in this game-theoretic setting. That is exactly why zero-shot and chain-of-thought are separate conditions in the grid rather than assumed inferior: if zero-shot matches Program-of-Thought on post-cutoff items, the method half of this proposal fails, and the benchmark half becomes the contribution.
Why not just stuff the domain definitions into the context (retrieval-augmented prompting)? TheoremQA already ran this experiment in spirit — "theorem augmentation," explicitly handing models the formal definitions, did not provide a significant boost, because the bottleneck was applying rules to dynamic problem states rather than recalling them Chen et al.. I am honest that the analogy is imperfect: AMM reserve math is more procedural than theorem application, so definition-stuffing might work better here than it did there. The grid therefore includes a definitions-in-context condition with no executable simulation, using the same retrieved corpus as the Program-of-Thought condition. If it matches Program-of-Thought on the inter-contract profitability items, the executable-simulation claim dies; if it fails there while Program-of-Thought succeeds, the TheoremQA finding extends to a genuinely adversarial, stateful domain.
Proposed Experiments and Decisive Checks
Experiment one measures the memorization delta. For each attack pattern class (displacement, insertion/sandwich, suppression), I construct matched pairs: a pre-cutoff instance the model has plausibly seen discussed, and a post-cutoff instance of the same structural class it cannot have. The headline metric is the paired accuracy drop across all reasoning conditions, reported per model with the cutoff date verified against each item's timestamp. A large delta confirms contamination inflation; a near-zero delta refutes it and, given the Chi et al. and Ravaut et al. evidence, would itself be a surprising result worth publishing.
Experiment two settles the reasoning-channel question on task family B, where ground truth is exact simulation. Metrics are profitability-decision accuracy, error in the computed unprofitability threshold $s_a$ relative to the true simulated value, and identification accuracy specifically on the inter-contract subset — the subset where Zhang et al. found 222 of 383 missed attacks concentrated Zhang et al.. Success for the Program-of-Thought claim is beating natural-language chain-of-thought by at least ten points on the inter-contract subset at matched token budget; anything smaller and the claim is not worth making.
Experiment three tracks degeneration over the stream's first year. Each quarter, the pre-cutoff-only knowledge synthesizer and the item-adaptive synthesizer are re-run on the newest items, giving a slope estimate per condition. The prediction, grounded in the temporal-degeneration findings of Zhu et al. and the mutating attack vectors documented in the MEV survey survey, is that static synthesis decays while adaptive synthesis holds. Experiment four reports the cost side: tokens-to-decision and wall-clock distributions per condition against a fixed block deadline, testing whether deliberation buys accuracy at a price that mempool-scale deployment cannot pay Si et al.. Together these four experiments either establish that current LLM auditing numbers are partly a mirage and that executable simulation plus temporal adaptation is the honest fix, or they establish the opposite on an instrument that will keep answering the question as long as the chain keeps producing blocks.
References
- Frontrunner Jones and the Raiders of the Dark Forest: An Empirical Study of Frontrunning on the Ethereum Blockchain
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution
- A flash(bot) in the pan: measuring maximal extractable value in private pools
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Combatting Front-Running in Smart Contracts: Attack Mining, Benchmark Construction and Vulnerability Detector Evaluation
- TheoremQA: A Theorem-driven Question Answering Dataset
- Eliminating Sandwich Attacks with the Help of Game Theory
- Competing Large Language Models in Multi-Agent Gaming Environments
- AiRacleX: Automated Detection of Price Oracle Manipulations via LLM-Driven Knowledge Mining and Prompt Generation
- ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs
- Maximal Extractable Value in Decentralized Finance: Taxonomy, Detection, and Mitigation
- Is Your LLM Outdated? A Deep Look at Temporal Generalization
- Excessive Reasoning Attack on Reasoning LLMs
- Advanced Smart Contract Vulnerability Detection via LLM-Powered Multi-Agent Systems
- A Comprehensive Survey of Contamination Detection Methods in Large Language Models
- Improving Low-resource Question Answering by Augmenting Question Information