DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization

Xueying Zhang, Yunjiang Jiang, Yue Shang, Zhaomeng Cheng, Chi Zhang, Xiaochuan Fan, Yun Xiao, Bo Long

Abstract

We propose DSGPT (Domain-Specific Generative Pre-training), a decoder-only Transformer model tailored for summarizing product titles and customer reviews in mobile e-commerce. Unlike general-purpose models that require massive datasets, DSGPT leverages a relatively small amount of domain-relevant pre-training data (the LCSTS dataset) to achieve state-of-the-art performance. Our approach eliminates the need for explicit background knowledge by implicitly capturing product attributes during training. Evaluations on Taobao and JD.com datasets demonstrate significant improvements in ROUGE scores and qualitative summary quality compared to existing methods.

Introduction: The Challenge of Mobile E-commerce

Mobile e-commerce platforms face a significant constraint: limited screen real estate. Sellers often populate product titles with numerous keywords to optimize search rankings, resulting in long, unreadable strings. Similarly, customer reviews can be verbose, making it difficult for users to quickly grasp key sentiments.

E-commerce mobile display showing product listings and review highlights.

Existing solutions often rely on large-scale models like BERT or GPT-3, which are resource-intensive and may lack the specific linguistic nuances of Chinese e-commerce. Furthermore, many state-of-the-art title summarization models require external databases of brand names and categories, which are difficult to maintain. DSGPT addresses these issues by focusing on domain-specific pre-training and an efficient decoder-only architecture.

Methods: Decoder-Only Architecture and Domain Pre-training

DSGPT utilizes a decoder-only Transformer architecture, which is particularly effective for text generation tasks. By treating the input and output as a single sequence, the model simplifies the fine-tuning process.

The DSGPT architecture: an auto-regressive decoder-only Transformer.

Domain-Specific Pre-training

Rather than training on a general corpus like Wikipedia, DSGPT is pre-trained on the LCSTS (Large-scale Chinese Short Text Summarization) dataset. This dataset, derived from Sina Weibo, contains 2.4 million training pairs. Pre-training on summarization-specific data allows the model to learn the fundamental logic of condensing information before being fine-tuned on e-commerce data.

Task-Specific Fine-tuning and Length Control

Following pre-training, the model is fine-tuned on specific datasets for title and review summarization. To optimize for mobile displays, the researchers introduced an automatic length control mechanism. This method applies an "amplification factor" to the <end> token during generation, encouraging the model to conclude the summary at a natural point that fits within display limits.

Datasets

The study employed three key datasets:

  1. LCSTS: Used for initial domain-specific pre-training.
  2. Taobao.com Title Summarization: A public dataset for fine-tuning the title summarization task.
  3. JD.com Review Summarization: An in-house dataset from JD.com used for fine-tuning the review highlight task.

Key Results and Insights

DSGPT was evaluated using ROUGE scores, a standard metric for summarization quality.

Title Summarization

On the Taobao dataset, DSGPT-large achieved a Rouge-L of 77.46, surpassing the previous state-of-the-art model (MS-Pointer_bi, 75.45). Notably, DSGPT achieved this without the explicit background knowledge required by the previous SOTA, demonstrating its ability to learn product attributes implicitly.

Review Summarization

On JD.com's internal data, DSGPT reached a Rouge-L of 76.73, outperforming standard Transformer models (74.50). The addition of the automatic length control method further boosted the Rouge-L score to 79.59.

Qualitative Performance

Case studies indicated that DSGPT is superior at preserving critical product descriptors (e.g., distinguishing between "girls" and "straight" clothing styles) that baseline models often omit.

Conclusion

DSGPT demonstrates that domain-specific pre-training on a targeted dataset is a highly effective and efficient alternative to massive general-purpose pre-training. By utilizing a decoder-only architecture and implicit knowledge capture, DSGPT provides a scalable solution for e-commerce text generation that is both high-performing and easier to maintain than previous methods. Future work may extend this framework to multi-modal summarization and other e-commerce domains.

Summary updated

Abstract

We propose a novel domain-specific generative pre-training (DS-GPT) method for text generation and apply it to the product titleand review summarization problems on E-commerce mobile display.First, we adopt a decoder-only transformer architecture, which fitswell for fine-tuning tasks by combining input and output all to-gether. Second, we demonstrate utilizing only small amount of pre-training data in related domains is powerful. Pre-training a languagemodel from a general corpus such as Wikipedia or the CommonCrawl requires tremendous time and resource commitment, andcan be wasteful if the downstream tasks are limited in variety. OurDSGPT is pre-trained on a limited dataset, the Chinese short textsummarization dataset (LCSTS). Third, our model does not requireproduct-related human-labeled data. For title summarization task,the state of art explicitly uses additional background knowledgein training and predicting stages. In contrast, our model implic-itly captures this knowledge and achieves significant improvementover other methods, after fine-tuning on the public Taobao.comdataset. For review summarization task, we utilize JD.com in-housedataset, and observe similar improvement over standard machinetranslation methods which lack the flexibility of fine-tuning. Ourproposed work can be simply extended to other domains for a widerange of text generation tasks.
BibTeX
@inproceedings{zhang2021dsgpt,
  author = {Xueying Zhang and Yunjiang Jiang and Yue Shang and Zhaomeng Cheng and Chi Zhang and Xiaochuan Fan and Yun Xiao and Bo Long},
  title = {{DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
  url = {https://openreview.net/forum?id=1BHjMM_sxMu7},
}
Figure