PALATE: Pretrained Autoregressive LATent Embeddings

Sizhang Zhao, Praveen Gondhi, Bin Li, Utsaw KumararXiv 2026 · arXiv

TL;DR

PALATE is a generative, GPT-style user representation framework developed at DoorDash that transforms sparse consumer interaction histories into dense, 256-dimensional embeddings. By using a two-stage curriculum and an exploration-weighted loss, it provides a single "frozen" feature that significantly boosts ads ranking performance across both browse and search surfaces, delivering substantial lifts in revenue and conversion rates.

What Problem Does This Paper Address?

In large-scale marketplaces like DoorDash, personalization is the "secret sauce" of ranking. However, it faces two major hurdles:

  1. High Cardinality and Sparsity: There are hundreds of thousands of stores, but any single consumer only interacts with a tiny fraction of them.
  2. Repurchase Bias: Consumer behavior is dominated by "favorites." Standard models often get "lazy," simply learning to predict the next repeat order rather than discovering new stores the user might like.

Previous production systems relied on hand-engineered aggregate counts (e.g., "clicks in the last 30 days") or order-agnostic "bags" of IDs. These methods fail to capture the nuanced sequential intent—the order and context of engagements—that modern Transformer architectures excel at.

Methods

The authors propose a generative approach to user modeling. Instead of building a complex sequence model inside every ranker, they train one standalone model to produce a reusable "user embedding."

The Architecture

The core is a decoder-only, GPT-style Transformer with three layers and three attention heads. It takes a chronological sequence of a consumer's engagements (store IDs, business IDs, and context like time of day) and predicts the next few businesses the consumer will engage with.

Figure 1: The generative user representation and how it is reused: a decoder-only GPT trained on each consumer’s store- engagement sequence, whose last-token hidden state is exported as a frozen 256-d feature and reused by the DCNv2 ranker across both ad surfaces.

Key Innovations

  • Two-Stage Curriculum: The model is first pretrained on plentiful click data to learn broad browsing behavior, then fine-tuned on conversion (purchase) data to sharpen its understanding of actual buying intent.
  • Exploration-Weighted Loss: To counter the repurchase-heavy nature of the data, the loss function up-weights predictions for new businesses (exploration) and down-weights repeats. This forces the model to learn what drives discovery.
  • Hard Negatives: During training, the model doesn't just look at random stores as negatives. It uses "hard negatives"—businesses the consumer was shown but chose not to order from—to force the representation to learn fine-grained preferences.
Figure 2: Model architecture. Per-position store, business, and context embeddings are concatenated (532-d), projected to a 768-d trunk, run through three pre-norm causal decoder blocks, and projected to 256-d. We export the last-token hid- den state as the user embedding and the softmax layer’s rows as business embeddings (§3.4).

Datasets

The model was trained and evaluated on DoorDash's internal production data, encompassing tens of millions of consumers and hundreds of thousands of businesses.

  • Training: Sequences of up to 200 historical engagements.
  • Evaluation: A held-out set from a time period following the training window to ensure no data leakage.

Key Insights and Ideas

  • The "Frozen" Advantage: By exporting the last-token hidden state as a frozen vector, DoorDash can run a single daily inference job to serve tens of millions of users. This is far more computationally efficient than running a Transformer inside the real-time ranking path for every ad request.
  • Weighting Matters: Without exploration weighting, the embedding provided almost zero gain in ranking. The 9:1 weighting (new vs. repeat) was the "aha moment" that made the representation useful for surfacing new merchants.
  • One Embedding to Rule Them All: Surprisingly, the same embedding improved performance on both Discovery (where users browse aimlessly) and Search (where users have high intent), proving it captures a fundamental "latent intent" of the consumer.

Key Results

PALATE became the largest single-feature gain in DoorDash's ranking-AUC ladder.

  • Offline Gains: It provided a relative Eval-AUC lift of +0.59% on Discovery and +0.57% on Search.
  • Online Impact: In A/B tests, shipping the embedding as part of a ranker update led to significant business metrics improvements:
  • Discovery: +0.79% Revenue, +1.17% Conversion Rate (CVR).
  • Search: +0.94% Revenue, +1.50% CVR.
Online A/B test results showing significant lifts across key metrics.

Limitations and Future Work

While highly effective, the authors note several areas for improvement:

  • Stale Intent: Because the embedding is refreshed daily, it misses "within-session" intent (e.g., if a user just clicked three pizza places in the last five minutes). A "live-session adapter" is a proposed future direction.
  • Missing Modalities: The current model doesn't incorporate text from search queries or geographic context (to avoid recommending a favorite burger joint that is currently out of delivery range).
  • Sequence Length: Future iterations may explore longer histories to capture more long-term user evolution.

Summary updated

Abstract

Local-commerce and food-delivery marketplaces run large ads businesses: across hundreds of thousands of stores, merchants pay to be surfaced on browse and search surfaces, and a ranker decides which ads each consumer sees. Ranking well demands strong per-consumer personalization, yet a consumer's raw interaction history is sparse and high-cardinality. Our production ranker had personalized only through hand-engineered signals: aggregate engagement counts, categorical embeddings, and order-agnostic pooled history, with no learned per-consumer sequence representation. We learn one by training a GPT-style generative model of the consumer's engagement sequence to predict their next few engagements, and read its last-token hidden state as the consumer embedding. That single frozen vector becomes one shared personalization input for every ad surface: a daily job refreshes it, and each ranker consumes it as an ordinary dense feature. Offline it is the largest single-feature gain in our ranking-AUC ladder, a relative Eval-AUC gain of +0.59% on a browse-driven feed and +0.57% on a query-driven search surface. Online, it shipped as one component of bundled ranker retrains on both surfaces: Discovery (revenue +0.79%, conversion rate +1.17%) and Search (revenue +0.94%, conversion rate +1.50%). Three design choices adapt the generative objective to a repurchase-heavy marketplace, and are the core of our contribution: an exploration-weighted loss that up-weights businesses new to the consumer over ones already engaged; a two-stage curriculum: a click pretrain that keeps the representation broad and exploratory, then a conversion fine-tune that sharpens purchase intent; and a designed negative set that pairs a fixed popular-business pool and businesses shown but not ordered with random and in-batch negatives.
BibTeX
@misc{zhao2026palate,
  author = {Sizhang Zhao and Praveen Gondhi and Bin Li and Utsaw Kumar},
  title = {{PALATE: Pretrained Autoregressive LATent Embeddings}},
  year = {2026},
  eprint = {2610.39090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39090},
}
Figure