MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems

Matthew Li, Albert Liang, Dilip RajarXiv 2026 · arXiv

TL;DR

Meta researchers have developed MLE Agents, autonomous LLM-based systems capable of proposing, implementing, and evaluating architectural changes for massive, industry-scale recommendation engines. By running over 1,000 experiments, the agents discovered novel model improvements that resulted in a +0.23% increase in search volume during a large-scale A/B test—demonstrating that autonomous agents can move the needle on systems processing billions of daily requests.

What Problem Does This Paper Address?

Improving large-scale ranking and recommendation models is a "slow and tedious" grind. Human engineers spend weeks forming hypotheses, writing code, and waiting for hours-long training jobs to finish, only to find that offline improvements often fail to translate into online user impact.

While LLM agents have shown promise on small-scale coding benchmarks, it remained unproven whether they could handle the "messiness" of industrial ML:

  1. Long Feedback Loops: Training takes hours, not seconds.
  2. Noisy Signals: Offline metrics (like AUPRC) often don't correlate with real-world user behavior.
  3. Saturation: Easy wins are found quickly, but finding the next 0.1% improvement requires deep architectural exploration.

Methods

The authors built a "human-on-the-loop" system where an LLM (Claude Opus 4.8) acts as the primary engineer, while humans provide high-level strategic direction.

The system evolved through two generations:

  • MLE Agents 1.0: A serial explorer that evaluated one candidate at a time. It taught the team that unstructured history leads to "prompt bloat" and that optimizing for the wrong metric leads to useless "wins."
  • MLE Agents 2.0: A parallelized system that runs seven candidates plus a "variance probe" (to measure training noise) simultaneously.

Key technical innovations include:

  • Calibrated Objective: A weighted composite of Normalized Entropy (NE) specifically tuned to match online engagement.
  • Context Engineering: Instead of feeding the agent the entire history, they used a "What Didn’t Work" blocklist to prevent the agent from repeating failed experiments and strict output contracts to reduce malformed code.
  • Architectural Sourcing: The agent was given access to summaries of external literature (e.g., papers on MaskNet, FiLM, and DHEN) to suggest genuinely new ideas rather than just tuning hyperparameters.

Datasets

The agents were tested on Meta's production search ranking surfaces:

  • Nullstate: Ranks query suggestions shown before a user starts typing.
  • Typeahead: Ranks suggestions as the user types.
  • Scale: The models are trained on hundreds of millions of examples and serve over a billion requests per day.

Key Insights and Ideas

  • The "Harness" is King: The researchers found that the design of the loop (the objective, the parallelism, the context) was more critical than the specific LLM used. A fixed model became significantly more productive simply by improving the "harness" around it.
  • Beyond Human Intuition: Surprisingly, the agent discovered architectural changes the human team had never tried, such as specific gating mechanisms and complex MLP structures drawn from broader literature.
  • The Saturation Curve: Improvements follow a "staircase" pattern. Early wins are frequent, but the curve flattens as the agent exhausts the obvious design space.
Figure 1: MLE agents 2.0 progress on Nullstate (weighted NE, lower is better): 1,097 distinct configurations, 325 kept. Best-so-far staircase annotated with the discoveries that moved it; the curve flattens after the early wins.
  • Failure Modes as Data: By logging "What Didn't Work," the system prevents "exploration collapse," where an agent might otherwise spiral into repeatedly trying to fix a failing optimizer setting.
Figure 4: Largest logged regressions on the “What Didn’t Work” blocklist (weighted NE degradation vs. baseline).

Key Results

  • Efficiency: In one campaign, the agent ran 1,097 completed experiments in 31 days, accepting 325 changes.
  • Win Rate: MLE Agents 2.0 achieved a 29.6% acceptance rate, nearly double the 15.6% rate of the 1.0 version.
  • Real-World Impact: The best agent-discovered model was deployed in a massive A/B test, yielding a +0.23% ± 0.15% increase in search session volume.
  • Diversity of Wins: As shown in Figure 3, the agent found wins across various categories, including loss weighting, DHEN capacity, and transformer architectures.
Figure 3: Completed experiments vs. accepted, by change category. Orange ticks mark the count expected by chance.

Limitations and Future Work

  • Exploration Collapse: Without human steering, agents can still get stuck in local minima (e.g., trying 8 variations of an optimizer that will never work).
  • Unreliable Self-Correction: The agent occasionally prunes its own good decisions or fails to revert bad ones consistently.
  • The Offline-Online Gap: While the calibrated objective helped, it isn't perfect. Some offline "winners" still fail to move the needle when they hit real users.
  • Future Direction: The authors suggest moving toward multi-agent teams and expanding the hypothesis space to include entirely new model surfaces. They argue that the role of the human engineer is shifting from "hypothesis executor" to "research director."

Summary updated

Abstract

The typical workflow for improving large-scale ranking and recommendation models is slow and tedious: an engineer forms a hypothesis, implements it as a code change, trains a model, and evaluates it both offline and online, often relying on their own intuition and experience along the way. Autonomous LLM agents promise to speed this up by running the loop on their own, but it is still unclear whether they can do this in industry-scale systems rather than on benchmarks or small proxy tasks. We build human-on-the-loop machine learning engineer (MLE) agents that propose, implement, train, and evaluate model changes on large-scale search rankers. Engineers retain control over objective design, search direction, and final-model review. We identify three obstacles that make this industry setting challenging: feedback takes hours per experiment, offline metrics do not always predict online impact, and improvements saturate after the early wins. We address these with parallel training runs, an objective calibrated to match online impact, and careful context engineering. In one campaign, the system ran 1,097 completed candidate experiments and accepted 325, a 29.6\% acceptance rate compared with 15.6\% in the earlier campaign under a different objective and search constraint. Its best model increases search volume by 0.23\% in an A/B test of large scale. Several improvements were architectural changes the team had never tried, drawn from the broader industry and literature, suggesting that MLE agents can explore regions of the design space that human intuition overlooks. We find that the design of the harness around the model matters: intuition suggests that a stronger underlying model is enough, but in our experience, the design of the loop around it is just as critical. More broadly, our work shows that autonomous agents can do real machine-learning engineering in industry-scale systems.
BibTeX
@misc{li2026mle,
  author = {Matthew Li and Albert Liang and Dilip Raj},
  title = {{MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems}},
  year = {2026},
  eprint = {2611.23910},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23910},
}
Figure