Data Quality Estimation Framework for Faster Tax Code Classification

Ravi Kondadadi, Allen Williams, Nicolas NicolovACL 2022

TL;DR

This paper introduces a novel framework, Data Quality Estimation (DQE), to automatically assess the quality of product descriptions for tax code classification. By combining attribute-value extraction and a quality assessment model, DQE helps identify and flag incomplete product data, significantly speeding up the customer onboarding process for tax compliance.

What Problem Does This Paper Address?

Businesses need to accurately map their product catalogs to tax codes for correct sales tax calculation. This process is currently very manual and time-consuming, largely due to poor quality product data. Common issues include:

  • Lack of Business Context: Product titles alone often don't provide enough information about the industry, forcing lengthy back-and-forth with customers.
  • Missing Attributes: Crucial product details (like material type for clothing) are frequently absent, making accurate tax code assignment difficult.
  • Ambiguous Language: The presence of rare words, acronyms, or abbreviations can confuse classification models.
  • Uncovered Industries: New or niche business sectors might not be well-represented in existing tax code taxonomies.

This paper aims to automate the identification of these data quality issues, thereby accelerating the onboarding process and improving the accuracy of tax code classification.

Methods

The proposed Data Quality Estimation (DQE) framework comprises three main components:

  1. Tax Code Classification Model: A tiered BERT-based model that first predicts a broad category and then a specific tax code for a product. This is a standard approach for text classification.
  2. Attribute-Value Extraction (AVE) Model: This is a novel application of Question Answering (QA) models. The product listing is treated as a document, and attribute names (derived from Avalara's tax code ontology) are used as questions to extract corresponding values. This approach is scalable and can handle unseen attributes. The authors experimented with fine-tuning DistilBERT and MiniLM models on the MAVE dataset for this task.
  3. Quality Assessment Model: A logistic regression model that uses features like prediction probabilities from the tax code classifier, identified missing attributes from the AVE model, title length, and category metadata to predict the quality of a product listing.

The overall framework iteratively refines the product data by identifying low-quality records, prompting users for missing information, and re-evaluating until a satisfactory quality level is achieved.

Datasets

  • MAVE Dataset: A large dataset (over 3 million attribute-value annotations from 2.2 million Amazon product descriptions) used for training the QA-based attribute-value extraction model.
  • Compliance Dataset: A smaller, manually annotated dataset of around 1,000 product listings, specifically curated for evaluating the framework's performance in a tax compliance context. This dataset includes attributes related to domains like insurance and medical care, which differ from the e-commerce focus of MAVE.

Key Insights and Ideas

  • QA for Attribute Extraction is Powerful: Framing attribute-value extraction as a Question Answering problem proves effective, especially for handling a wide range of attributes and unseen ones, outperforming traditional rule-based or lookup methods.
  • Combining Signals for Quality: Simply relying on tax code classification probabilities isn't enough for robust data quality assessment. Incorporating missing attribute information (from AVE) and other features significantly improves the accuracy of identifying problematic product listings.
  • Iterative Refinement is Key: The framework's iterative nature, where it prompts users for missing information and re-evaluates, is crucial for achieving high data quality and completing the onboarding process efficiently.
  • Domain-Specific Data is Important: While models trained on large e-commerce datasets (like MAVE) perform well, performance dips on domain-specific datasets (like the Compliance dataset) highlight the need for domain-aligned training data.

Key Results

  • Attribute-Value Extraction: The QA-based models (AVE-QA-DISTILBERT and AVE-QA-MINILM) significantly outperformed the AVE-FT (hybrid lookup/classification) approach on both the MAVE and Compliance datasets, achieving F1-scores of 0.54-0.63 on the Compliance dataset. MiniLM showed a slight edge over DistilBERT.
  • Data Quality Estimation: The proposed logistic regression model for quality assessment, which incorporates prediction probabilities, missing attributes, and other features, achieved a higher F1-score (0.828) compared to baselines that relied solely on prediction probabilities (0.755) or prediction probabilities plus tax-code precision (0.741).
  • Feature Importance: Title length and the presence of missing attributes were identified as the most important features for quality estimation.

Limitations and Future Work

  • Performance on Compliance Data: The authors acknowledge that the current performance of the AVE models on the Compliance dataset is still low. They plan to augment the MAVE dataset with compliance-related information and retrain the models.
  • Generalizability: While the framework is designed to be applicable to services and utilities, further work is needed to adapt it to other tax code ontologies like the Harmonized Commodity Description and Coding System (HS).
  • User Interaction: The paper outlines an iterative process involving user feedback, but the specifics of this interaction and its efficiency are areas for continued development.
  • Production Integration: The authors are currently working on integrating this tool into production to estimate data quality for customer product catalogs.

Summary updated

BibTeX
@inproceedings{kondadadi2022data,
  author = {Ravi Kondadadi and Allen Williams and Nicolas Nicolov},
  title = {{Data Quality Estimation Framework for Faster Tax Code Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure