TL;DR
Meta researchers have developed an end-to-end "agentic" system that uses Large Language Models (LLMs) to automatically optimize feature selection for massive fleets of ranking models. By treating the LLM as a "Researcher" that proposes and verifies hypotheses within an experiment loop, the system balances model accuracy against strict hardware constraints and shared infrastructure costs, leading to significant gains in both offline metrics and real-world online performance.
What Problem Does This Paper Address?
In large-scale web platforms, ranking models (like those used for ads or recommendations) don't exist in a vacuum. They operate as a "fleet," drawing from a shared pool of tens of thousands of potential features.
ML engineers traditionally face a grueling manual trade-off: adding more features can improve prediction accuracy (measured by Normalized Entropy, or NE), but it also increases costs. These costs are two-fold:
- Serving Costs: Per-model constraints like latency, memory, and throughput (Queries Per Second, or QPS).
- Pipeline Costs: Shared infrastructure costs for data processing, storage, and logging.
Optimizing a single model is hard enough; optimizing a whole fleet while accounting for shared costs—where a feature used by Model A is essentially "free" for Model B to adopt—is a combinatorial nightmare for human engineers.
Methods
The authors propose an agentic system that automates the feature lifecycle through four main stages:
- Agentic Feature Creation: An LLM acts as a retrieval engine to propose "cross-features" (e.g., combining
user_countryandad_category). This generator is continually fine-tuned on successful outcomes from across the entire fleet, meaning the system gets smarter as it optimizes more models. -
Feature Ranking & Blending: To handle multi-task models (e.g., predicting both clicks and conversions), the system uses Pareto Optimality. Instead of manually weighting different tasks, it selects features that reside on the Pareto front, ensuring that a feature is kept if it strongly benefits even a single task. Figure 1b: Feature blending using Pareto fronts to balance importance across multiple tasks.
-
The LLM Researcher Agent: This is the core "brain" of the system. It runs an observe-reason-act loop. It looks at the history of experimental trials, reasons about which feature categories (dense, sparse, etc.) to expand or trim, and proposes new configurations (hypotheses) to test.
- Fleet-Aware Trimming: Before a trial runs, a deterministic tool checks if the proposed features exceed the shared pipeline budget. If they do, it "trims" the most expensive new features and replaces them with high-performing features already in use by other models in the fleet.
The overall architecture is summarized in the system overview below: Figure 1a: The end-to-end agentic system overview, showing the loop between the LLM agent and deterministic tools.
Datasets
The system was evaluated using Meta’s internal production ads ranking models and a fleet-wide "Signal" database containing metadata and importance scores for tens of thousands of features. Evaluation was performed via:
- Offline Experiments: Training on production data windows to measure Normalized Entropy (NE) and resource usage (FLOPs, QPS).
- Online A/B Tests: Deploying optimized models to live traffic to measure primary business metrics.
Key Insights and Ideas
- The "Free" Feature Insight: The most novel aspect is the fleet-level awareness. By recognizing that pipeline costs are shared, the system incentivizes models to converge on a high-quality shared feature set, reducing the overall infrastructure footprint.
- LLM as an Optimizer, Not Just a Coder: Unlike systems that use LLMs just to write feature engineering code, this system uses the LLM to manage the strategy of experimentation—deciding when to be greedy, when to explore, and how to evolve previous successful trials.
- Rebalancing vs. Scaling: The researchers found that gains often come from rebalancing volumes across feature categories (e.g., swapping expensive sparse features for more efficient embedding features) rather than simply adding more of everything.
Key Results
The system demonstrated that an automated agent can match or beat manual engineering across a diverse fleet:
- Accuracy: Improved Normalized Entropy (NE) by 0.31% on average across production models within their respective resource budgets.
- Efficiency: The trimming step reduced attributed pipeline costs by 2.3% (with some models seeing up to 6.3% savings) without regressing model quality.
- Online Impact: In live A/B testing, the optimized models delivered a 0.20% improvement in a primary online metric.
- Search Effectiveness: As shown in the search progress plots, the LLM agent consistently found "faster and better" models (lower NE and higher QPS) over the course of 25 trials. Figure 2: Search progress showing how the agent finds new "best" candidates (rings) across different behavior families.
Limitations and Future Work
- Mixture Tuning: While the system can tune the "mixture" of different feature importance estimators, this specific capability was not fully evaluated in the current paper.
- Search Space Constraints: The current search is largely focused on per-category volumes. Expanding the agent's agency to modify model architecture or hyperparameters alongside features is a logical next step.
- Reasoning Evaluation: The "Researcher" agent provides free-text rationales for its actions, but the paper does not quantitatively evaluate the "correctness" of this reasoning, only the final performance of the proposed trials.
Summary updated