Small firms are often denied credit because lenders cannot see their finances. They have no audited statements and no bureau history, only raw records such as bank-statement transactions or court filings. Sequence models, meaning networks that read these records one event at a time, can turn such records into credit signals. The difficulty is that labeled SME data is tiny and has few defaults. The one public-facing bank-statement study in this area has 611 applicants and 93 defaults. Practitioners therefore reach for synthetic data: they fit a generator to the few real histories they have, sample many more, and train or evaluate on those samples. This proposal asks whether that practice works when the downstream model is a recurrent neural network (RNN), a network that carries a running memory $h_t$ forward through a sequence. It also asks how we would know whether it works.
The core idea fits in two sentences. A recurrent model is flexible enough to detect the fingerprints a generator leaves in event timing and ordering, so it may score well on synthetic data because it has learned the generator, not the borrower. That failure should be larger for sequence models than for tree models on summary statistics. We test this in a simulated SME world where the true answer is known. From that test we derive a small set of temporal fidelity checks, together with cross-generator evaluation, that predict when synthetic histories can safely stand in for real ones.
Core Claim
Two findings in the literature have never been put side by side. The first is that sequence representations beat engineered aggregates in credit. E.T.-RNN feeds raw transactions to a GRU ensemble, a gated recurrent network, and reaches 0.83 ROC AUC against 0.81 for LightGBM built on about 7,000 hand-crafted features. Learned Sequence Representations over Raw Credit Events raises AUPR from 0.068 to 0.147 by replacing aggregates with learned sequence embeddings. The second finding is that synthetic data evaluated with a single generator is overconfident. Synthetic Data, Real Errors shows a naive estimate of 0.841 AUC against a true 0.738, and fixes much of this with a Deep Generative Ensemble (DGE): K independently trained generators, with models trained on one generator's output and evaluated on the others'.
Our claim is that these findings interact. The same flexibility that lets a recurrent model beat aggregates on real data also lets it overfit generator artifacts. We predict three things. First, the naive synthetic-estimate gap is larger for recurrent encoders than for tree models on aggregates. Second, DGE-style cross-evaluation closes more of that gap for recurrent encoders. Third, the static fidelity tiers of Structured Evaluation of Synthetic Tabular Data, which cover marginals, pairwise, leave-one-out conditionals, and joint distributions, fail to predict downstream error for recurrent encoders. Temporal tiers do predict it: inter-event time distributions, autocorrelation decay, and next-event conditionals. If any of these three predictions fails, we learn that synthetic sequence data is either safer or more useless than practice assumes. Either result would be useful to anyone underwriting thin-file firms.
This differs from the closest work in specific ways. DGE was tested on static tabular and image data, never on sequence encoders. Yang et al.'s tiers are explicitly static. On the Equivalency, Substitutability, and Flexibility of Synthetic Data measures a synthetic-to-real "exchange rate" (about 30 synthetic frames per real frame for tracking), but in vision, where the synthetic engine is hand-built rather than fitted to scarce data. Credit papers that use synthetic data, such as WSMOTE-ensemble and deep generative reject inference, work on static records and report downstream AUC from a single generative fit.
Research Program
The first component is a ground-truth SME simulator, which motivates the rest. Real SME data cannot tell us the "true" performance of a model trained on synthetic data, because a real test set of 244 firms with roughly 37 defaults gives AUC estimates too noisy to separate the effects we care about. We borrow the logic of Is this Model for Real?, which simulates data from a known process to check whether a fitted model recovers reality. Our simulator gives each firm a latent health trajectory. That trajectory drives irregularly timed inflows and outflows with merchant-like categories, seasonal cycles, and occasional shocks. Default follows specified mechanisms, including slow balance erosion, abrupt loss of a major customer, and rising volatility.
The simulator is calibrated to match the published cash-flow summaries of AI-BAAM: six months of statements, about 15% defaults, and features such as balance volatility and log growth. Its default mechanisms echo the supply-chain dependence found by Yang et al.'s ST-GNN work. We draw "real" training cohorts of AI-BAAM size from it, and we draw a large fresh population for testing, so true AUROC and AUPR are known to high precision. Several variants inject mechanisms that are rare in the training draw, to probe tail behavior. Deliberate temporal drift probes the warning in Hinder et al.'s drift survey.
The second component is a real anchor. The SMEsD data from Combining intra-risk and contagion risk contains 3,976 Chinese SMEs with 11,523 timestamped lawsuits from 2014 to 2021, labeled for bankruptcy. Per-firm lawsuit histories are irregular event sequences, which is exactly the setting where the fixed time-step assumption of recurrent updates breaks. We use this dataset to check that the simulator's conclusions about ranking and gaps hold on real data, accepting a noisier test set. If access to a bank-statement cohort like AI-BAAM's can be negotiated, it becomes a second anchor. It is proprietary, and we do not assume access.
The third component is temporal fidelity tiers. We extend Yang et al.'s static hierarchy with three sequence-level tiers. The first compares per-class distributions of inter-event gaps and event counts. The second compares autocorrelation decay of balance and flow series. The third is a next-event conditional tier: a recurrent model trained on real sequences scores the likelihood of synthetic ones, and a model trained on synthetic sequences scores real ones, in the leave-one-out spirit of Yang et al. but conditioned on history. We compute these tiers per class, because the default class is where scarce-data generators fail.
The fourth component is a modular harness. It has three swappable parts: a generator, an encoder, and an evaluator. The generator takes real sequences and returns K synthetic cohorts. The encoder is either a sequence model or aggregates plus trees. The evaluator runs naive, DGE, and true-population evaluation. The design follows the .fit/.predict contract of the Universal Causal Evaluation Engine and the pre-training, fusion, and task split of UniTS.
Modularity is here for a methodological reason, not a software one. The file notes that preprocessing choices in temporal pipelines leak into encoders, for example time binning or gap imputation. If the generator sees binned data and the encoder sees raw timestamps, part of the measured gap is a pipeline artifact. The harness fixes preprocessing as a shared module and lets us run that leak as an explicit ablation. Code and simulator are released so others can plug in their own generators.
The generators are an autoregressive GRU that samples the next event type, amount, and time gap; a marked point-process-style model with a parametric intensity; a within-class block bootstrap of real sequences as the simplest generator; and, for the aggregate pipeline, a CART-based tabular synthesizer on flattened features. Yang et al. found synthesizers of this CART type strongest on small tables. The encoders are a GRU with the time gap as an input, a GRU with learned time embeddings, a temporal convolutional network, and gradient-boosted trees on aggregates. This spread follows Clements et al., whose TCN trained about 40% faster than their LSTM at 92.33% Gini. Handling irregular gaps as an input follows Razavian et al. on irregular clinical trajectories.
Two quantities organize the results. The estimation gap for a model is $\Delta = \widehat{A}_{\text{syn}} - A_{\text{true}}$. Here $\widehat{A}_{\text{syn}}$ is its AUPR estimated on synthetic data, naively or by DGE cross-evaluation, and $A_{\text{true}}$ is its AUPR on the fresh simulated population. The exchange rate is
$$\rho(m) = \min \{ s : A_{\text{true}}(n + s_{\text{syn}}) \ge A_{\text{true}}(n + m_{\text{real}}) \} / m,$$
that is, how many synthetic firms are worth one additional real firm, starting from a real cohort of size $n$. An infinite $\rho$ means synthetic data never substitutes, whatever its volume.
Baselines
The do-nothing reference trains each encoder on the real cohort alone, with no synthetic data, and evaluates it by repeated real cross-validation. In the simulator we also know its true performance. Every synthetic method must beat this on true AUPR to justify itself. The closest published reference point for a thin-file SME setting is logistic regression on application and bank-statement features in AI-BAAM: 0.806 AUROC on its validation set (AI-BAAM). We reproduce that model class on simulator cohorts calibrated to the same size and default rate. We do not claim the absolute number transfers.
The strongest competitor for predictive accuracy is gradient-boosted trees on engineered aggregates. We run it both with and without monotone constraints, since Intuit's small-business study reports that monotone XGBoost improves K-S by 8% over unconstrained XGBoost on noisy banking data. For sequence-versus-aggregate comparisons, the reported numbers are E.T.-RNN's 0.83 ROC AUC against LightGBM's 0.81 on proprietary consumer data. At SME scale we expect trees to be harder to beat than those numbers suggest. For SMEsD, the file gives no reported numeric result for ComRisk: no reported result found on Lacuna for SMEsD AUC. We therefore rerun a tree baseline on litigation aggregates ourselves.
The strongest synthetic-data competitor for imbalance is WSMOTE-style oversampling of aggregates with a tree ensemble. It reports 0.9910 AUC on 3,111 Chinese small-business records (WSMOTE-ensemble). That number came from a single synthetic fit on static features, so we treat it as a method to rerun, not a target to match.
The strongest evaluation competitor is DGE applied as published, on flattened aggregates. It reports 0.724 against a true 0.738, versus a naive 0.841 (Synthetic Data, Real Errors).
All methods get the same real cohort, the same preprocessing module, the same hyperparameter search budget in GPU-hours, and the same K when ensembles are used. Tuning uses only real-cohort cross-validation, never the true population.
Beating the baselines means two things. First, a synthetic-augmented recurrent model must exceed the best real-only model, whether tree or sequence, on true AUPR and on recall at a fixed low decline rate, with confidence intervals from at least 20 simulator seeds. Second, our temporal tiers must predict $\Delta$ and $\rho$ better than Yang et al.'s static tiers, measured by rank correlation across generator-encoder pairs.
Three ablations isolate the key ingredient. One trains the recurrent encoder on synthetic sequences with inter-event times randomly permuted within each firm. If timing artifacts drive congeniality, this should shrink $\Delta$. A second replaces DGE cross-evaluation with a single generator. A third deliberately mismatches preprocessing between generator and encoder to measure the leak.
Why not just…?
Why not just use gradient-boosted trees on aggregates and skip synthetic sequences? This is a real threat at 611 firms. The sequence-over-tree evidence in E.T.-RNN and Learned Sequence Representations comes from 145,000 to 740,000 consumers, not small SME cohorts. We do not argue around it. The exchange-rate curves settle the question directly. If trees on real aggregates match the best synthetic-augmented recurrent model on true AUPR at every cohort size, the answer for thin-file lenders is to use trees and skip synthetic sequences, and we report that.
Why not just flatten sequences into tables and use existing DGE and Yang et al. metrics? Flattening throws away exactly the timing and ordering information a recurrent model could exploit, whether as real signal or as generator artifact. Static metrics on aggregates would score a generator well even if it produced implausible gap patterns. The time-permutation ablation and the static-versus-temporal tier correlation test whether this matters. If static tiers predict $\Delta$ as well as temporal ones, the temporal extension is unnecessary.
Why not just hold out real data and test on it (train-synthetic, test-real)? With about 37 defaults in a 40% validation split, AUPR estimates have confidence intervals wide enough to hide gaps of the size DGE reports. That is why the simulator exists: it gives true performance with negligible noise. On SMEsD we do report train-synthetic, test-real, but we treat it as a coarse consistency check.
Why not just use a faster non-recurrent encoder such as a TCN? We include one. The question is whether flexible sequence encoders in general absorb generator artifacts. Comparing the GRU with the TCN tells us whether recurrence specifically matters, or only access to raw sequence.
Proposed Experiments and Decisive Checks
The main experiment crosses four generators, four encoders, cohort sizes from 300 to 3,000 simulated firms, and three evaluation modes (naive, DGE with K up to 20, and true population), over 20 simulator seeds and the tail-mechanism and drift variants. We report true AUROC, AUPR, and recall at a 0.5% and a 5% decline rate. We also report $\Delta$ under each evaluation mode, Kendall rank agreement between estimated and true model rankings, and $\rho(m)$. We then repeat the generator-encoder grid on SMEsD lawsuit sequences with train-synthetic, test-real evaluation. The check there is whether the simulator's rankings of generators and encoders reappear.
Four results decide the proposal. The central prediction fails if the naive $\Delta$ for recurrent encoders is not larger than for aggregate trees, since that would mean sequence models are no more vulnerable to generator artifacts. The mechanism is supported only if permuting inter-event times in synthetic training data shrinks the recurrent model's $\Delta$ without much loss of true AUPR. The temporal tiers earn their place only if they rank generator-encoder pairs by true downstream error clearly better than the static tiers. Synthetic sequences are worth using for thin-file SME credit only if $\rho$ is finite and the best synthetic-augmented model beats real-only trees at AI-BAAM scale. If that last check fails, the proposal still delivers something useful: a tested warning, with a reusable harness and simulator, that fitted synthetic histories should not be used to train or grade sequence credit models at SME scale.
References
- E.T.-RNN: Applying Deep Learning to Credit Loan Applications
- Learned Sequence Representations over Raw Credit Events for Credit-Abuse Scoring
- Synthetic Data, Real Errors: How (Not) to Publish and Use Synthetic Data
- Structured Evaluation of Synthetic Tabular Data
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Combining weighted SMOTE with ensemble learning for the class-imbalanced prediction of small business credit risk
- Deep generative models for reject inference in credit scoring
- Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality
- AI-BAAM: AI-Driven Bank Statement Analytics as Alternative Data for Malaysian MSME Credit Scoring
- Financial Risk Analysis for SMEs with Graph-based Supply Chain Mining
- One or two things we know about concept drift - a survey on monitoring in evolving environments. Part A: detecting concept drift
- Combining intra-risk and contagion risk for enterprise bankruptcy prediction using graph neural networks
- Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models
- UniTS: A Universal Time Series Analysis Framework Powered by Self-Supervised Representation Learning
- Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data
- Multi-task Prediction of Disease Onsets from Longitudinal Laboratory Tests
- Using Small Business Banking Data for Explainable Credit Risk Scoring