TL;DR
This paper evaluates whether LLM agents can function as "Forward-Deployed Engineers" (FDEs) by navigating the messy, real-world process of requirement discovery and system delivery for enterprise text-to-SQL. While top-tier models are surprisingly good at gathering requirements through dialogue, the end-to-end success rate for delivering a production-ready system remains low (25%), revealing a significant gap between passing academic benchmarks and meeting strict enterprise constraints.
What Problem Does This Paper Address?
Most LLM coding benchmarks focus on isolated tasks with clear specifications (e.g., "write a function that does X"). However, real-world engineering—especially the role of a Forward-Deployed Engineer (FDE)—is much more complex. An FDE must embed with a customer, turn ambiguous business requests into technical requirements, and then build a system that respects hardware, cost, and latency constraints.
The authors tackle the question: Can LLM agents handle this full lifecycle? They focus on enterprise text-to-SQL, where a model must translate natural language into database queries. This isn't just about accuracy; it involves specific SQL dialects (Oracle), strict latency targets (under 15s), and hardware limitations (CPU-only serving).
Methods
The researchers developed a framework grounded in a real four-month human FDE engagement at Oracle. The process is split into two phases:
- Discovery Phase: An agent interacts with a "simulated customer" (powered by Gemini 2.5 Pro) to uncover 17 core requirements and 6 explicit unknowns. The agent starts with varying levels of ambiguity (Low, Moderate, Extreme) and must ask follow-up questions to fill a "Fact Sheet."
- Delivery Phase: Using the discovered requirements, the agent must plan and execute a post-training recipe (e.g., fine-tuning a 7B-8B model using LoRA). The authors compared two workflows:
- Decoupled: A "Planner" agent creates a structured experiment plan, which is then frozen and handed to an "Execution" agent.
- Autonomous: A single agent plans and executes iteratively in one session.
Datasets
- Training/Evaluation: The study used Oracle-dialect variants of standard benchmarks: Spider, BIRD, and SParC.
- Production Workload: A newly introduced set of 100 production questions obtained from a real enterprise customer to test "production readiness."
- Ground Truth: A human-validated "Fact Sheet" derived from the original enterprise engagement, used to score the agent's discovery performance.
Key Insights and Ideas
- The "Discovery Loss": Agents often fail not because they didn't ask the right questions, but because they failed to record the answers in the final handoff. About 21% of facts disclosed by the customer never made it into the agents' final requirement documents.
- Benchmark vs. Production Gap: Passing a development benchmark like Spider does not guarantee production success. Several agents hit accuracy targets on public datasets but failed on the production workload due to SQL compilation errors or latency spikes.
- Poor Budget Management: Agents struggled with GPU "budgeting." They often underutilized available compute (e.g., using only 5 hours of a 12-hour window) or misallocated time between training and evaluation.
- Targeted Clarification Wins: Success in discovery depends more on asking high-quality, targeted questions than on the length of the initial customer brief.
Key Results
- Discovery Performance: Frontier models like Claude 5.5 and Claude 5 led the pack, recovering over 80% of withheld requirements. Open-weight models (e.g., Llama 3.3, Gemma 4) struggled, recovering only 40-50%.
- Delivery Success: Only 25% of the delivery runs (2 out of 8) met all three acceptance criteria: accuracy, production compilation validity, and latency.
- Human Baseline: The human FDE still outperformed all agents, achieving higher accuracy (79.9% Spider EX) and better production validity than the best agentic run.
Limitations and Future Work
The study is a bounded case study of a single enterprise engagement, which may not represent the full diversity of customer communication styles. Additionally, the evaluation relies on "LLM-as-a-judge," which can introduce its own biases.
The authors suggest that for LLM agents to become truly viable FDEs, we need:
- Human-in-the-loop: Reviewing discovered requirements and experimental plans before execution.
- Explicit Environment Specs: Better "harnesses" that provide agents with clear credentials, hardware constraints, and budget scopes.
- Production-Centric Evaluation: Moving beyond public benchmarks to include production-specific checks like SQL compilation rates on real-world schemas.
Summary updated