Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data

Haitham Hammami, Louis Baligand, Bojan PetrovskiNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of accurately parsing unstructured address data in financial payment messages, a crucial step for regulatory compliance. They introduce a new, noisy, real-world dataset and demonstrate that fine-tuned Transformer models, particularly XLM-RoBERTa-Large with early stopping, achieve state-of-the-art performance, outperforming existing tools like LibPostal and DeepParse.

What Problem Does This Paper Address?

Financial institutions face a significant challenge in identifying the origin and destination of funds due to the prevalence of unstructured address data in payment messages. While newer standards like ISO 20022 aim for structured addresses, a substantial volume of messages still relies on free-text formats. This necessitates robust address parsing to extract critical fields like street, postal code, and country, which is essential for regulatory compliance. Existing solutions often struggle with the "noise" inherent in real-world transactional data, such as misspellings, missing information, or extraneous details.

Methods

The authors explore several approaches to address parsing:

  • Transformer Models: They investigate various Transformer-based models, including distilled versions of BERT (DistilBERT, DistilRoBERTa) and larger models like BERT-base and XLM-RoBERTa-Large. They experiment with different training strategies, including early stopping and modifications to input features (e.g., removing "HardSep" tokens or prefixes).
  • Decoder-Based Approaches (LLMs): They evaluate large language models (LLMs) like Llama 2 and Mistral-7B, using carefully crafted prompts to extract address information. They also explore fine-tuning these models with LoRA adapters.
  • Baseline Models: LibPostal, a popular open-source address parser, and DeepParse, a Seq2Seq bidirectional-LSTM model, are used as baselines for comparison.

Datasets

The paper introduces a new, augmented dataset designed to mimic real-world noisy payment data.

  • Synthetic Data (Vo): This is the base dataset derived from DeepParse's training data, consisting of clean, structured addresses.
  • Augmented Data (V1 & V2):
    • V1: Involves basic cleaning and address structure masking (rearrangement or removal of elements).
  • V2: Incorporates more extensive augmentation, including adding names, country information, simulating line separations, and introducing "Out-Of-Address" (OOA) elements to mimic real-world noise.
  • Production Data: A manually labeled set of 1,600 messages was used to guide the data augmentation process and for final evaluation.

Key Insights and Ideas

  • The importance of noise: Real-world payment data is inherently noisy, and models that can handle this noise significantly outperform those trained on clean data.
  • Transformer superiority: Fine-tuned Transformer models, especially larger ones like XLM-RoBERTa-Large, demonstrate superior performance and robustness compared to traditional methods and even smaller Transformer variants.
  • Early stopping is crucial: For Transformer models, employing early stopping based on zero-shot data performance is a key technique to prevent overfitting and improve generalization.
  • LLMs show potential but have limitations: While generative LLMs can perform address parsing with prompt engineering, they are more susceptible to hallucinations and output format inconsistencies, making them less reliable for this specific task in a production environment without further refinement.
  • Data augmentation is key: The V2 augmented dataset, which closely mimics real-world noise, is critical for training robust models and achieving better zero-shot performance.

Key Results

  • The XLM-RoBERTa-Large model, fine-tuned with early stopping on the V2 dataset, achieved the highest F1 scores across all datasets, including the production data (0.801 on production data).
  • DistilBERT models generally outperformed DeepParse across various data versions.
  • LibPostal showed consistent performance but struggled with the V2 dataset due to its structured training data and inability to handle OOA elements.
  • For LLMs, Mistral-7B (especially fine-tuned versions) showed promising zero-shot performance, with F1 scores reaching up to 0.7233 for Mixtral 8x7B Instruct v0.1.

Limitations and Future Work

  • LLM Hallucinations: The authors noted that LLMs can sometimes hallucinate, modify word indices, or produce unexpected output formats (e.g., nested JSON), requiring post-processing.
  • Country Derivation: While the paper focuses on address parsing, explicitly deriving the country from addresses lacking explicit country information remains a challenge.
  • Future Directions: The authors suggest exploring context-free grammars (like LMQL) for more structured LLM output and integrating geography knowledge with tools like LlamaIndex. They also plan to open-source their code, models, and dataset to facilitate further research.

Summary updated

BibTeX
@inproceedings{hammami2024fighting,
  author = {Haitham Hammami and Louis Baligand and Bojan Petrovski},
  title = {{Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-industry.17},
}
Figure