Cost-Aware Agentic Feature Optimization System for Fleet-Scale Ranking Models

Chang Yang, Yao Xuan, Wendan Yan, Nhat Le, Jiang Liu, Longhao Jin, Zhen Peggy Yao, Alireza Vahdatpour, Djordje GligorijevicarXiv 2026 · arXiv

TL;DR

Meta researchers have developed an end-to-end "agentic" system that uses Large Language Models (LLMs) to automatically optimize feature selection for massive fleets of ranking models. By treating the LLM as a "Researcher" that proposes and verifies hypotheses within an experiment loop, the system balances model accuracy against strict hardware constraints and shared infrastructure costs, leading to significant gains in both offline metrics and real-world online performance.

What Problem Does This Paper Address?

In large-scale web platforms, ranking models (like those used for ads or recommendations) don't exist in a vacuum. They operate as a "fleet," drawing from a shared pool of tens of thousands of potential features.

ML engineers traditionally face a grueling manual trade-off: adding more features can improve prediction accuracy (measured by Normalized Entropy, or NE), but it also increases costs. These costs are two-fold:

  1. Serving Costs: Per-model constraints like latency, memory, and throughput (Queries Per Second, or QPS).
  2. Pipeline Costs: Shared infrastructure costs for data processing, storage, and logging.

Optimizing a single model is hard enough; optimizing a whole fleet while accounting for shared costs—where a feature used by Model A is essentially "free" for Model B to adopt—is a combinatorial nightmare for human engineers.

Methods

The authors propose an agentic system that automates the feature lifecycle through four main stages:

  1. Agentic Feature Creation: An LLM acts as a retrieval engine to propose "cross-features" (e.g., combining user_country and ad_category). This generator is continually fine-tuned on successful outcomes from across the entire fleet, meaning the system gets smarter as it optimizes more models.
  2. Feature Ranking & Blending: To handle multi-task models (e.g., predicting both clicks and conversions), the system uses Pareto Optimality. Instead of manually weighting different tasks, it selects features that reside on the Pareto front, ensuring that a feature is kept if it strongly benefits even a single task. Figure 1b: Feature blending using Pareto fronts to balance importance across multiple tasks.

  3. The LLM Researcher Agent: This is the core "brain" of the system. It runs an observe-reason-act loop. It looks at the history of experimental trials, reasons about which feature categories (dense, sparse, etc.) to expand or trim, and proposes new configurations (hypotheses) to test.

  4. Fleet-Aware Trimming: Before a trial runs, a deterministic tool checks if the proposed features exceed the shared pipeline budget. If they do, it "trims" the most expensive new features and replaces them with high-performing features already in use by other models in the fleet.
Figure 1: (a) The constrained search loops under resource caps until the trial budget 𝐾is spent. (b) Feature blending takes whole Pareto fronts, outermost first, until 𝑁features are selected; A and B share a front, and A dominates C.

The overall architecture is summarized in the system overview below: Figure 1a: The end-to-end agentic system overview, showing the loop between the LLM agent and deterministic tools.

Figure 1a: The end-to-end agentic system overview, showing the loop between the LLM agent and deterministic tools.

Datasets

The system was evaluated using Meta’s internal production ads ranking models and a fleet-wide "Signal" database containing metadata and importance scores for tens of thousands of features. Evaluation was performed via:

  • Offline Experiments: Training on production data windows to measure Normalized Entropy (NE) and resource usage (FLOPs, QPS).
  • Online A/B Tests: Deploying optimized models to live traffic to measure primary business metrics.

Key Insights and Ideas

  • The "Free" Feature Insight: The most novel aspect is the fleet-level awareness. By recognizing that pipeline costs are shared, the system incentivizes models to converge on a high-quality shared feature set, reducing the overall infrastructure footprint.
  • LLM as an Optimizer, Not Just a Coder: Unlike systems that use LLMs just to write feature engineering code, this system uses the LLM to manage the strategy of experimentation—deciding when to be greedy, when to explore, and how to evolve previous successful trials.
  • Rebalancing vs. Scaling: The researchers found that gains often come from rebalancing volumes across feature categories (e.g., swapping expensive sparse features for more efficient embedding features) rather than simply adding more of everything.

Key Results

The system demonstrated that an automated agent can match or beat manual engineering across a diverse fleet:

  • Accuracy: Improved Normalized Entropy (NE) by 0.31% on average across production models within their respective resource budgets.
  • Efficiency: The trimming step reduced attributed pipeline costs by 2.3% (with some models seeing up to 6.3% savings) without regressing model quality.
  • Online Impact: In live A/B testing, the optimized models delivered a 0.20% improvement in a primary online metric.
  • Search Effectiveness: As shown in the search progress plots, the LLM agent consistently found "faster and better" models (lower NE and higher QPS) over the course of 25 trials. Figure 2: Search progress showing how the agent finds new "best" candidates (rings) across different behavior families.
Figure 2: Feature search (25 trials). (a) Error bars: ±1 s.d. of QPS over 3 runs. (b) Marker shape and color give the behavior family; rings mark new bests.

Limitations and Future Work

  • Mixture Tuning: While the system can tune the "mixture" of different feature importance estimators, this specific capability was not fully evaluated in the current paper.
  • Search Space Constraints: The current search is largely focused on per-category volumes. Expanding the agent's agency to modify model architecture or hyperparameters alongside features is a logical next step.
  • Reasoning Evaluation: The "Researcher" agent provides free-text rationales for its actions, but the paper does not quantitatively evaluate the "correctness" of this reasoning, only the final performance of the proposed trials.

Summary updated

Abstract

Large Web platforms operate large fleets of ranking models, each selecting features from a shared pool of tens of thousands. We present an end-to-end agentic system that optimizes features across the fleet under resource constraints shared across models and those set per model, such as feature pipeline cost and serving throughput (QPS), respectively. In this system, LLM agents propose hypotheses, and deterministic tools execute experiments and verify them. For each model, an LLM generates and shortlists cross features from a combinatorial candidate space, and all features are then ranked by several importance methods. Feature blending mixes these rankings into a single selection, using Pareto fronts across tasks for multi-task models. An LLM Researcher agent then tunes the feature selection in 10--25 asynchronous trials under resource caps, and a trimming step keeps the pipeline cost within budget. Across production models, the system improves normalized entropy by 0.31% on average within each model's budget, and the trimming step cuts pipeline cost by 2.3% with no regression beyond measurement noise. The optimized models were launched, and in online A/B tests they improved a primary online metric by 0.20%.
BibTeX
@misc{yang2026cost,
  author = {Chang Yang and Yao Xuan and Wendan Yan and Nhat Le and Jiang Liu and Longhao Jin and Zhen Peggy Yao and Alireza Vahdatpour and Djordje Gligorijevic},
  title = {{Cost-Aware Agentic Feature Optimization System for Fleet-Scale Ranking Models}},
  year = {2026},
  eprint = {2611.26531},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26531},
}
Figure