Abstract
We propose DSGPT (Domain-Specific Generative Pre-training), a decoder-only Transformer model tailored for summarizing product titles and customer reviews in mobile e-commerce. Unlike general-purpose models that require massive datasets, DSGPT leverages a relatively small amount of domain-relevant pre-training data (the LCSTS dataset) to achieve state-of-the-art performance. Our approach eliminates the need for explicit background knowledge by implicitly capturing product attributes during training. Evaluations on Taobao and JD.com datasets demonstrate significant improvements in ROUGE scores and qualitative summary quality compared to existing methods.
Introduction: The Challenge of Mobile E-commerce
Mobile e-commerce platforms face a significant constraint: limited screen real estate. Sellers often populate product titles with numerous keywords to optimize search rankings, resulting in long, unreadable strings. Similarly, customer reviews can be verbose, making it difficult for users to quickly grasp key sentiments.
Existing solutions often rely on large-scale models like BERT or GPT-3, which are resource-intensive and may lack the specific linguistic nuances of Chinese e-commerce. Furthermore, many state-of-the-art title summarization models require external databases of brand names and categories, which are difficult to maintain. DSGPT addresses these issues by focusing on domain-specific pre-training and an efficient decoder-only architecture.
Methods: Decoder-Only Architecture and Domain Pre-training
DSGPT utilizes a decoder-only Transformer architecture, which is particularly effective for text generation tasks. By treating the input and output as a single sequence, the model simplifies the fine-tuning process.
Domain-Specific Pre-training
Rather than training on a general corpus like Wikipedia, DSGPT is pre-trained on the LCSTS (Large-scale Chinese Short Text Summarization) dataset. This dataset, derived from Sina Weibo, contains 2.4 million training pairs. Pre-training on summarization-specific data allows the model to learn the fundamental logic of condensing information before being fine-tuned on e-commerce data.
Task-Specific Fine-tuning and Length Control
Following pre-training, the model is fine-tuned on specific datasets for title and review summarization. To optimize for mobile displays, the researchers introduced an automatic length control mechanism. This method applies an "amplification factor" to the <end> token during generation, encouraging the model to conclude the summary at a natural point that fits within display limits.
Datasets
The study employed three key datasets:
- LCSTS: Used for initial domain-specific pre-training.
- Taobao.com Title Summarization: A public dataset for fine-tuning the title summarization task.
- JD.com Review Summarization: An in-house dataset from JD.com used for fine-tuning the review highlight task.
Key Results and Insights
DSGPT was evaluated using ROUGE scores, a standard metric for summarization quality.
Title Summarization
On the Taobao dataset, DSGPT-large achieved a Rouge-L of 77.46, surpassing the previous state-of-the-art model (MS-Pointer_bi, 75.45). Notably, DSGPT achieved this without the explicit background knowledge required by the previous SOTA, demonstrating its ability to learn product attributes implicitly.
Review Summarization
On JD.com's internal data, DSGPT reached a Rouge-L of 76.73, outperforming standard Transformer models (74.50). The addition of the automatic length control method further boosted the Rouge-L score to 79.59.
Qualitative Performance
Case studies indicated that DSGPT is superior at preserving critical product descriptors (e.g., distinguishing between "girls" and "straight" clothing styles) that baseline models often omit.
Conclusion
DSGPT demonstrates that domain-specific pre-training on a targeted dataset is a highly effective and efficient alternative to massive general-purpose pre-training. By utilizing a decoder-only architecture and implicit knowledge capture, DSGPT provides a scalable solution for e-commerce text generation that is both high-performing and easier to maintain than previous methods. Future work may extend this framework to multi-modal summarization and other e-commerce domains.
Summary updated