Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL

Marianne Menglin Liu, Rongguang Wang, Zheng Wang, Yewen Wang, Leonid Boytsov, Tao Sheng, Sujith Ravi, Dan RotharXiv 2026 · arXiv

TL;DR

This paper evaluates whether LLM agents can function as "Forward-Deployed Engineers" (FDEs) by navigating the messy, real-world process of requirement discovery and system delivery for enterprise text-to-SQL. While top-tier models are surprisingly good at gathering requirements through dialogue, the end-to-end success rate for delivering a production-ready system remains low (25%), revealing a significant gap between passing academic benchmarks and meeting strict enterprise constraints.

What Problem Does This Paper Address?

Most LLM coding benchmarks focus on isolated tasks with clear specifications (e.g., "write a function that does X"). However, real-world engineering—especially the role of a Forward-Deployed Engineer (FDE)—is much more complex. An FDE must embed with a customer, turn ambiguous business requests into technical requirements, and then build a system that respects hardware, cost, and latency constraints.

The authors tackle the question: Can LLM agents handle this full lifecycle? They focus on enterprise text-to-SQL, where a model must translate natural language into database queries. This isn't just about accuracy; it involves specific SQL dialects (Oracle), strict latency targets (under 15s), and hardware limitations (CPU-only serving).

Methods

The researchers developed a framework grounded in a real four-month human FDE engagement at Oracle. The process is split into two phases:

  1. Discovery Phase: An agent interacts with a "simulated customer" (powered by Gemini 2.5 Pro) to uncover 17 core requirements and 6 explicit unknowns. The agent starts with varying levels of ambiguity (Low, Moderate, Extreme) and must ask follow-up questions to fill a "Fact Sheet."
  2. Delivery Phase: Using the discovered requirements, the agent must plan and execute a post-training recipe (e.g., fine-tuning a 7B-8B model using LoRA). The authors compared two workflows:
  3. Decoupled: A "Planner" agent creates a structured experiment plan, which is then frozen and handed to an "Execution" agent.
  4. Autonomous: A single agent plans and executes iteratively in one session.
Figure 1: A forward deployment agent: discovery, then delivery. In discovery, a discovery agent turns the ambiguous request of a customer simulated from a real engagement into identified requirements (1)–(4), establishes by experiment what the customer cannot answer (the SQL dialect), and gets sign-off. In delivery, a planner agent turns the identified requirements into an experiment plan and an execution agent runs it, profiling on the target hardware and evaluating on (1)–(4), returning model artifacts and an accuracy, profiling and cost report.

Datasets

  • Training/Evaluation: The study used Oracle-dialect variants of standard benchmarks: Spider, BIRD, and SParC.
  • Production Workload: A newly introduced set of 100 production questions obtained from a real enterprise customer to test "production readiness."
  • Ground Truth: A human-validated "Fact Sheet" derived from the original enterprise engagement, used to score the agent's discovery performance.

Key Insights and Ideas

  • The "Discovery Loss": Agents often fail not because they didn't ask the right questions, but because they failed to record the answers in the final handoff. About 21% of facts disclosed by the customer never made it into the agents' final requirement documents.
  • Benchmark vs. Production Gap: Passing a development benchmark like Spider does not guarantee production success. Several agents hit accuracy targets on public datasets but failed on the production workload due to SQL compilation errors or latency spikes.
  • Poor Budget Management: Agents struggled with GPU "budgeting." They often underutilized available compute (e.g., using only 5 hours of a 12-hour window) or misallocated time between training and evaluation.
  • Targeted Clarification Wins: Success in discovery depends more on asking high-quality, targeted questions than on the length of the initial customer brief.

Key Results

  • Discovery Performance: Frontier models like Claude 5.5 and Claude 5 led the pack, recovering over 80% of withheld requirements. Open-weight models (e.g., Llama 3.3, Gemma 4) struggled, recovering only 40-50%.
Table 1: Discovery results averaged over 10 independent runs per model and ambiguity level. Each entry reports the mean (top) and standard deviation (bottom). Bold indicates the highest mean in each recovery rate and elicitation count.
  • Delivery Success: Only 25% of the delivery runs (2 out of 8) met all three acceptance criteria: accuracy, production compilation validity, and latency.
  • Human Baseline: The human FDE still outperformed all agents, achieving higher accuracy (79.9% Spider EX) and better production validity than the best agentic run.
Table 2: End-to-end delivery results; each agent runs in both decoupled and autonomous workflows on OpenCode. CPU serving uses Q4_K_M GGUF base models (4-bit-class mixed weight quantization) with F16 LoRA adapters (16-bit floating point). EX is execution accuracy (%), the higher, the better; “Prod. fail” is the SQL compilation failure rate on production SQLs, the lower, the better; p90 is CPU inference latency, the lower, the better. The human row is the FDE’s shipped system under the same evaluator.

Limitations and Future Work

The study is a bounded case study of a single enterprise engagement, which may not represent the full diversity of customer communication styles. Additionally, the evaluation relies on "LLM-as-a-judge," which can introduce its own biases.

The authors suggest that for LLM agents to become truly viable FDEs, we need:

  1. Human-in-the-loop: Reviewing discovered requirements and experimental plans before execution.
  2. Explicit Environment Specs: Better "harnesses" that provide agents with clear credentials, hardware constraints, and budget scopes.
  3. Production-Centric Evaluation: Moving beyond public benchmarks to include production-specific checks like SQL compilation rates on real-world schemas.

Summary updated

Abstract

LLM coding agents can increasingly automate model post-training, but whether they can act as forward-deployed engineer (FDE) agents, discovering customer requirements and turning them into deployment decisions, remains unclear. We study this on a real enterprise text-to-SQL engagement: a simulated customer, grounded in the real use case's requirements, starts with an ambiguous request. An agent must discover acceptance criteria, data permissions, and hardware constraints in the discovery phase, then plan and execute a post-training recipe in the delivery phase. Across 15 models, the strongest discover over 80% of withheld requirements and open-weight models discover 40-50%; in the delivery phase, only 25% of runs across eight coding-agent configurations pass all acceptance criteria. We identify four bottlenecks in this engagement: (1) incomplete discovery, with 40% of withheld requirements missing or misrepresented; (2) unreliable execution; (3) underused compute budgets and a lack of systematic comparisons across base models and post-training methods; and (4) benchmark accuracy does not guarantee production readiness. We suggest human review of discovered requirements, monitoring of the agent's plan during execution, and an explicit experiment-environment specification covering hardware, budget scope, and production checks.
BibTeX
@misc{liu2026can,
  author = {Marianne Menglin Liu and Rongguang Wang and Zheng Wang and Yewen Wang and Leonid Boytsov and Tao Sheng and Sujith Ravi and Dan Roth},
  title = {{Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL}},
  year = {2026},
  eprint = {2611.03551},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03551},
}
Figure