Introduction
In the evolving landscape of digital retail, "agentic commerce" represents a shift from passive search bars to autonomous AI assistants capable of product discovery, catalog navigation, and transaction completion. PayPal, a leader in global payments, recently collaborated with NVIDIA to enhance its first-generation Commerce Agent. The primary objective was to overcome significant performance bottlenecks inherent in using general-purpose Large Language Models (LLMs) for specialized commerce tasks.
The Search and Discovery Bottleneck
PayPal's initial analysis revealed that the "Search and Discovery" component was responsible for over 50% of the total system response time. While traditional keyword-based search is insufficient for natural language intent, massive off-the-shelf LLMs introduced unacceptable latency—often exceeding 4.5 seconds—and high operational costs. To address this, the team sought a "Goldilocks" solution: a Small Language Model (SLM) that maintained high reasoning quality while operating within strict latency and cost constraints.
Methodology: Fine-Tuning for Commerce
The core of the optimization strategy involved leveraging the NVIDIA NeMo Framework to specialize the llama3.1-nemotron-nano-8B-v1 architecture. The researchers employed a multi-stage optimization pipeline:
- Parameter-Efficient Fine-Tuning (PEFT): Using Low-Rank Adaptation (LoRA), the model was adapted to commerce-specific nuances without the prohibitive cost of full parameter retraining.
- Supervised Fine-Tuning (SFT): The model was trained on 10,000 high-quality prompt-response pairs synthesized from real customer chat interactions.
- Direct Preference Optimization (DPO): A reinforcement learning-style technique was applied using a preference-based dataset of 7,457 rows to refine the model's judgment between "better" and "worse" responses.
- Hypothetical Document Embeddings (HyDE): The SLM was trained to generate ideal product descriptions based on user queries, which were then used to query the actual product catalog, bridging the gap between natural language and structured data.
Agentic Orchestration and Infrastructure
Beyond model weights, the system's success relied on a robust orchestration layer. The Agent Orchestrator manages short-term and long-term memory, allowing the assistant to maintain context across a shopping session. This is integrated with a personalization engine and a product catalog to ensure recommendations are both relevant and accurate.
To maximize throughput, the system was benchmarked on NVIDIA H100 and B200 (Blackwell) GPUs. Utilizing 3D parallelism (Tensor, Pipeline, and Data parallelism) through NVIDIA NIM, the team achieved a 35% latency improvement from hardware upgrades alone, complementing the software-level optimizations.
Key Results and Impact
The transition to the fine-tuned Nemotron SLM yielded dramatic improvements in production metrics:
- Latency Reduction: Overall agent latency dropped by 49%, with retrieval-specific latency improving by 58%. This brought the system to a sub-2-second response target.
- Cost Efficiency: Monthly GPU operational costs were reduced by 45% compared to the previous LLM-based architecture.
- Quality Gains: Despite the smaller parameter count, the SFT-tuned model achieved a 23% improvement in quality score over the non-fine-tuned 8B baseline, as measured by "LLM-as-a-Judge" metrics.
Conclusion and Future Work
This collaboration demonstrates that domain-specific specialization can allow smaller models to outperform larger, general-purpose counterparts in production environments. By focusing on the query formulation stage—the primary bottleneck in retrieval-heavy systems—PayPal successfully balanced performance, cost, and quality. Future efforts will focus on extending these techniques to other specialized agents within the PayPal ecosystem, including post-purchase support and personalized ranking systems.
Summary updated