TL;DR
The Large Payments Model (LPM) is a domain-specific foundation model designed to serve as a "reusable intelligence layer" for structured payment data. By pretraining on up to 400 billion tokens of transaction records, LPM achieves massive performance gains—up to an 85% uplift in fraud detection—while meeting the strict sub-60ms latency requirements of global financial systems.
What Problem Does This Paper Address?
In the payments industry, tasks like fraud detection, authorization prediction, and payment optimization are usually handled by independent, siloed models. This is inefficient because these tasks rely on overlapping information about transaction structure and entity behavior.
However, building a "foundation model" for payments is harder than building one for text. Payment records are highly structured, containing a mix of categorical IDs (with millions of unique values), continuous amounts, and timestamps. Furthermore, these models must respect the "operational lifecycle"—they cannot use information that wouldn't be available at the exact moment a transaction is authorized. Existing Large Language Models (LLMs) struggle with this native structure, and standard tabular models often fail to capture long-term temporal dependencies or scale to billions of records.
Methods
The authors introduce two complementary architectures for LPM, both utilizing a Field-Aware Encoder that maps heterogeneous inputs (numerics, text, and high-cardinality IDs) into a common representation space.
- The Independent Variant: Designed for maximum efficiency, this model processes transactions individually using fixed-size aggregate features for historical context. It uses a hybrid masking strategy during pretraining that combines random masks with "lifecycle-aware" masks to ensure the model doesn't "cheat" by looking at future outcomes.
- The Sequence Variant: This model captures rich temporal context by processing entity-linked transaction histories. It uses an interleaved attention pattern (alternating between intra-transaction and inter-transaction blocks) to efficiently model dependencies across time without the computational explosion typically seen in long-sequence transformers.
A standout technical detail is the use of non-uniform partitioned embeddings for high-cardinality identifiers (like card tokens or merchant IDs). Instead of a massive, uniform embedding table, LPM allocates more capacity to frequent/important values and less to the "long tail," keeping the model size manageable.
Datasets
The models were trained on massive, proprietary datasets from JPMorgan Chase:
- Merchant Services: Card transactions spanning credit, debit, and prepaid rails.
- Treasury Services: Automated Clearing House (ACH) and wire transfers.
- Scale: Pretraining corpora ranged from 18 billion to 400 billion tokens, constructed using entity-aware stratified sampling to ensure rare but operationally critical patterns were represented.
Key Insights and Ideas
- Structured Data as a First-Class Modality: Rather than serializing transactions into text for an LLM, LPM treats the fixed schema as a feature. This allows for specialized numerical encodings (using Fourier and piecewise-linear branches) that capture non-linear patterns like "sharp" spending thresholds.
- The Value of Sequence: The researchers found that while simple aggregate features are sufficient for some tasks, the Sequence variant provided an additional 23 percentage point uplift in fraud detection, proving that raw temporal patterns contain signals that manual feature engineering misses.
- Zero-Shot-ish Adaptation: A frozen LPM backbone can be adapted to entirely new tasks (e.g., predicting if a declined transaction will eventually be approved) using a simple linear probe or LoRA. This confirms that the model learns a generalized "language of payments" rather than just memorizing specific labels.
Key Results
LPM demonstrated significant improvements over strong, task-specific production baselines:
- Fraud Detection: The Independent model saw a 62.7% uplift in Average Precision (AP), while the Sequence model reached 85.5%.
- Network Response Prediction: Fine-grained prediction of ~100 different response codes improved by 80-89% in macro-F1 score.
- Treasury Services: ACH fraud prevention saw a 27% uplift in volume-based recall and a 63.5% uplift in value-based recall.
- Operational Efficiency: In production, the system handles millions of transactions daily with a p99 latency under 60ms.
Limitations and Future Work
- Fixed Schemas: The current architecture relies on a fixed schema. Extending the model to new fields or different domains currently requires significant retraining.
- Bounded Context: While the Sequence model is powerful, it uses a bounded history (e.g., the last 32 transactions). Capturing "infinite" or very long-term entity memory remains an open challenge.
- Cross-Domain Learning: Currently, Merchant and Treasury models are trained separately. The authors suggest that a single, unified cross-payment representation is the next logical step.
- Generative Use-Cases: The paper focuses on discriminative tasks (prediction). Using LPM for generative tasks, such as synthetic transaction generation for privacy-preserving research, was not explored.
Summary updated