Does Named Entity Recognition Truly Not Scale Up to Real-world Product Attribute Extraction?

Wei-Te Chen, Keiji Shinzato, Naoki Yoshinaga, Yandi XiaEMNLP 2023

TL;DR

This paper re-evaluates the scalability of Named Entity Recognition (NER) for extracting product attributes from e-commerce sites, challenging the notion that Question Answering (QA)-based approaches are inherently superior. The authors demonstrate that BERT-based NER models can achieve comparable accuracy to BERT-based QA models on a large-scale, real-world dataset, while offering significantly faster inference times, making them a more practical solution for handling millions of products.

What Problem Does This Paper Address?

Extracting product attributes (like "color," "size," "material") from e-commerce product descriptions is crucial for improving search and recommendation systems. This task, known as Attribute-Value Extraction (AVE), is challenging due to the sheer volume and diversity of attributes (potentially thousands) and products. Traditionally, AVE has been framed as a sequence tagging problem, similar to Named Entity Recognition (NER). However, recent research has shifted towards Question Answering (QA)-based approaches, which frame attribute extraction as answering a question about a specific attribute. These QA models have shown promise, but the authors argue that previous comparisons to NER models were unfair, often using weak baselines and limited evaluation settings. This paper aims to provide a more rigorous comparison to determine if NER can truly scale to real-world AVE challenges.

Methods

The core of this paper is a comparative study of two main approaches for AVE:

  • BERT-NER: This approach uses a BERT model as a powerful text encoder, followed by a feed-forward layer to classify each token into a specific attribute tag (e.g., "B-Color" for the beginning of a color attribute, "I-Color" for inside, "O" for outside). This allows the model to extract values for multiple attributes simultaneously.
  • BERT-QA: This approach also leverages BERT but is designed to extract values for a single target attribute at a time. It takes both the product text and the target attribute as input and predicts the start and end positions of the attribute value. To handle multiple attributes, the QA model needs to be run multiple times for each product.

The authors implemented a modified BERT-QA model that can extract multiple values for a given attribute by replacing its decoder with a feed-forward layer, similar to the BERT-NER approach. They also adopted the BILOU (Begin, Inside, Last, Outside, Unit) chunk tag scheme for both models.

Datasets

The primary dataset used in this study is the MAVE dataset (Yang et al., 2022). This dataset is a large-scale, real-world AVE benchmark derived from 2.2 million products from Amazon Review Data. It includes product text from various sources (title, description, features) and annotations for attribute values, including "NONE" for attributes not mentioned.

To address potential biases in the original MAVE dataset that might favor QA models (e.g., by providing few target attributes per example), the authors also created a subset called TestNER. This subset specifically excludes examples where the BERT-NER model incorrectly predicts attributes outside the target set or where values for multiple attributes overlap, aiming for a fairer comparison.

Key Insights and Ideas

  • Unfair Comparisons in Prior Work: The authors argue that previous studies claiming QA's superiority over NER for AVE were based on unfair comparisons, often using outdated NER baselines (like BiLSTM trained from scratch) and not leveraging the full potential of modern pre-trained models like BERT for NER.
  • BERT-NER's Scalability: Contrary to previous assumptions, BERT-based NER models can scale to a large number of attributes (up to a thousand) in real-world AVE scenarios, achieving competitive accuracy.
  • QA's Advantage is Context-Dependent: The perceived advantage of QA models often stems from their ability to handle overlapping attribute values and avoid predicting irrelevant attributes by explicitly focusing on a single target attribute. However, these issues can be mitigated in NER models with appropriate techniques.
  • Inference Speed Matters: For large-scale e-commerce applications with millions of products, inference speed is a critical factor. BERT-NER significantly outperforms BERT-QA in terms of inference time because it processes the text once for all attributes, whereas QA models require multiple passes for each attribute.

Key Results

  • Accuracy: On the MAVE dataset, BERT-NER achieved comparable accuracy to BERT-QA, especially on the TestNER subset designed for a fairer comparison. For instance, on TestNER, BERT-NER achieved an F1 score of 92.58, while BERT-QA achieved 93.87. The difference was even smaller when considering only examples without overlapping attributes.
  • Inference Time: BERT-NER was significantly faster than BERT-QA. On the test set, BERT-NER took 905 seconds, while BERT-QA took 1464 seconds. This gap widens as the number of attributes increases.
  • Attribute Handling: While QA models naturally handle overlapping values and avoid irrelevant attribute predictions by design, the authors suggest that NER models can address these issues through techniques like limiting the output tag space or using nested NER approaches.

Limitations and Future Work

  • Overlapping Attributes: The paper acknowledges that QA models have an advantage in handling overlapping attribute values, a scenario where NER models might struggle without specialized techniques.
  • Irrelevant Attribute Prediction: While BERT-NER can be made to avoid predicting irrelevant attributes by limiting the output tag space, this requires prior knowledge of the target attributes.
  • Future Work: The authors suggest revisiting NER-based approaches as a robust and practical solution for AVE. They also point to the need for further research into mitigating the challenges of overlapping attributes and irrelevant predictions within the NER framework. They also highlight the importance of a comprehensive attribute taxonomy for QA models to be efficient.

Summary updated

BibTeX
@inproceedings{chen2023does,
  author = {Wei-Te Chen and Keiji Shinzato and Naoki Yoshinaga and Yandi Xia},
  title = {{Does Named Entity Recognition Truly Not Scale Up to Real-world Product Attribute Extraction?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure