Can Large Language Models Optimize Pretrained Neural Network Weights? A Multi-Phase Empirical Evaluation of Gradient-Free Editing in YOLO26 Object Detection

Mohamed Abdallah SalemarXiv 2026 · arXiv

TL;DR

Can an LLM "reason" its way into better neural network weights without using gradients? This multi-phase study finds that while LLMs can propose syntactically valid weight edits for YOLO26 detectors, they offer no statistically significant advantage over simple random-policy controls, and the tiny gains achieved rarely generalize to unseen data.

What Problem Does This Paper Address?

Adapting a pretrained model usually requires gradient-based fine-tuning—a process that is computationally intensive and requires access to the original training pipeline. A tempting alternative is Weight-Space Programming (WSP): directly editing the model's parameters (weights) to improve performance on a specific task or dataset.

However, the search space for weight edits is astronomical. The authors investigate whether Large Language Models (LLMs), acting as "reasoning" optimizers, can guide these edits better than traditional black-box methods like Differential Evolution or CMA-ES. They specifically tackle the question of whether LLM-guided interventions can fix specific failure modes (like false positives) in a YOLO26 object detector.

Methods

The researchers implemented a closed-loop framework to test various gradient-free editing strategies across five experimental phases:

  1. Direct Interventions: Simple scaling and stochastic perturbations to establish a baseline for parameter sensitivity.
  2. Numerical Optimizers: Using Differential Evolution (DE) and CMA-ES to search for optimal weight modifications.
  3. LLM-Guided Proposals: Using a local LLM (Gemma via Ollama) to propose edits. The LLM was provided with failure diagnostics (e.g., "too many false positives for 'dog'") and asked to suggest which layers to modify and by how much.
  4. Hybrid Approaches: A "LLM + CMA-ES" hybrid where the LLM selects the target layers (the "where") and CMA-ES optimizes the numerical coefficients (the "how much").
  5. Rigorous Controls: Crucially, the authors compared LLM strategies against a matched random-policy control—a baseline that receives the same numerical search budget but lacks the LLM's "semantic" reasoning.

![Fig.

  1. The Direct Weight-Space Programming framework used in the study.](/figures/art_5ca978a80dd14d5caca761338f75d2b0)

Datasets

The study utilized the Microsoft COCO Val2017 dataset (5,000 images), partitioned into:

  • Development Set (1,000 images): Used as the objective function for the search process.
  • Selection Set (1,500 images): A disjoint set used to measure how well the "best" edits from the development set actually generalize.
  • Holdout Set (2,500 images): Used for final confirmatory evaluations in early phases.

Key Insights and Ideas

  • The "Selection Effect" Trap: It is relatively easy to find weight edits that improve scores on a small development set. However, these gains almost always vanish (or turn negative) when tested on a disjoint selection set. This suggests the optimizers are often just "fitting the noise" of the small sample.
  • Sensitivity is Non-Uniform: The detection head of YOLO26 is significantly more sensitive to weight changes than the backbone or neck stages. Small tweaks to the head can lead to massive drops in accuracy, whereas the backbone is more robust (see Figure 2).
  • Interpretability $\neq$ Utility: While an LLM can provide a very convincing rationale for why it is editing a specific layer (e.g., "adjusting Stage 8 to reduce localization errors"), these "reasoned" edits didn't perform better than random layer selections in controlled tests.
  • The Importance of Strict Validation: A forensic audit revealed that many early "successful" LLM edits were actually just local Gaussian steps. When a strict schema was enforced, the LLM's rejection rate skyrocketed (172 rejections vs. 53 accepted proposals).

![Fig.

  1. Empirical sensitivity scores showing that the detection head (S23) is far more sensitive to edits than the backbone.](/figures/art_ae73085d582a4afb9eceafa69962b310)

Key Results

  • CMA-ES Scaling: Increasing the search budget for CMA-ES from 8 to 250 evaluations improved development gains (+0.11 to +0.28 points) but resulted in worse performance on the selection set (-0.03 to -0.12 points).
  • LLM vs. Random: In head-to-head comparisons at budgets of 8, 25, and 50 evaluations, the LLM hybrid showed no statistically significant advantage over the random-policy hybrid (p-values of 0.612, 0.369, and 0.694, respectively).
  • YOLO26m Replication: A five-seed replication on the larger YOLO26m model yielded a negligible official COCO API AP change of just +0.004 percentage points.
  • Computational Overhead: LLM-guided search is significantly slower and more resource-intensive due to token generation and inference time, without providing a commensurate boost in accuracy.

![Fig.

  1. Comparison of different optimizers over 15 evaluations. While some show quick gains, they fail to establish a consistent lead.](/figures/art_81b484762b16485f8e0e415384ab61b8)

Limitations and Future Work

The authors are refreshingly candid about the limitations:

  • Search Budgets: The experiments used relatively small search budgets (up to 250 evaluations), which might be insufficient for complex weight-space landscapes.
  • Generalization Scope: The results are specific to YOLO26 and COCO; different architectures or tasks might respond differently to weight editing.
  • Provenance Issues: The study notes inconsistencies in how some data subsets were handled in early phases, highlighting the need for better "forensic" tracking in LLM-based research.

Future Work: The authors recommend that future studies on LLM-guided optimization must use preregistered policies, frozen architecture maps, and genuinely untouched test data to avoid the pervasive "selection effects" documented here. They argue that the burden of proof remains on showing that LLM "reasoning" provides any incremental value over numerical baselines in weight space.

Summary updated

Abstract

Direct weight editing offers a gradient-free alternative to fine-tuning pretrained detectors, but its generalization and the value of large language model (LLM) guidance remain uncertain. We evaluate five experimental phases using YOLO26 models and MS COCO, comparing deterministic scaling, random perturbations, differential evolution, CMA-ES, structured interventions, real local-LLM proposals, failure-aware guidance, and hybrid LLM-CMA-ES against a matched random-policy control. Optimization uses native Ultralytics mAP50-95 on 1,000 COCO Val2017 images; selected candidates are assessed on a disjoint, previously examined 1,500-image subset, with COCO API AP50-95 reported separately where available. Increasing the CMA-ES budget from 8 to 250 evaluations raised development gains from +0.11 to +0.28 percentage points, while selection changes remained negative (-0.03 to -0.12 points); the 250-evaluation result uses one seed. The hybrid LLM approach showed no statistically established advantage over random-policy guidance (p = 0.612, 0.369, and 0.694 at budgets 8, 25, and 50). Five-seed YOLO26m replication yielded a COCO API AP difference of +0.004 points (95% CI: -0.026 to +0.034). Of 225 archived LLM proposals, 53 satisfy the final strict schema; early tensor-level execution cannot be fully verified. Genuine LLM-mediated weight editing is feasible, but the tested approaches did not demonstrate consistent transfer beyond the development subset or a measurable LLM advantage.
BibTeX
@misc{salem2026can,
  author = {Mohamed Abdallah Salem},
  title = {{Can Large Language Models Optimize Pretrained Neural Network Weights? A Multi-Phase Empirical Evaluation of Gradient-Free Editing in YOLO26 Object Detection}},
  year = {2026},
  eprint = {2611.17789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17789},
}
Figure