TL;DR
DSGPT (Domain-Specific Generative Pre-training) is a decoder-only Transformer model designed to summarize e-commerce titles and reviews for mobile screens. By pre-training on a relatively small, domain-relevant Chinese dataset rather than a massive general corpus, the model achieves state-of-the-art results while remaining efficient enough for real-world deployment.
What Problem Does This Paper Address?
In the world of mobile e-commerce, screen real estate is at a premium. Sellers often create "keyword-stuffed" product titles (sometimes exceeding 70 characters) to game search algorithms, making them unreadable for humans on small screens. Similarly, customer reviews are often too long to display effectively in search results.
While large-scale models like GPT-3 or BERT can handle text generation, they face three major hurdles in this context:
- Resource Intensity: Pre-training on general corpora like Wikipedia or Common Crawl is incredibly expensive and often wasteful if the end goal is a specific niche like "Chinese e-commerce."
- Language/Domain Gap: Many open-source models are English-centric or lack the specific "shorthand" logic required for product titles.
- Complexity: Previous state-of-the-art (SOTA) models for title summarization relied on external "background knowledge" (like explicit databases of brand names) to ensure accuracy, which is difficult to maintain and integrate.
Methods
The researchers from JD.com proposed a two-stage approach using a decoder-only Transformer architecture (similar to the GPT family).
- The Architecture: They chose a decoder-only structure because it simplifies the pipeline—input and output are treated as a single sequence. This makes the model easier to scale and more flexible for different tasks by simply adding special tokens.
- Domain-Specific Pre-training: Instead of training on the entire internet, they pre-trained DSGPT on the LCSTS dataset (Chinese microblogging summaries). This taught the model the general "logic" of summarization in Chinese before it ever saw a product title.
- Task-Specific Fine-tuning: The model was then fine-tuned on specific e-commerce datasets.
- Automatic Length Control: To ensure summaries weren't just cut off mid-sentence, they introduced a generation method that applies an "amplification factor" to the
<end>token, encouraging the model to find a natural stopping point that fits mobile display constraints.
Datasets
The study utilized three primary datasets:
- LCSTS (Pre-training): A large-scale Chinese short text summarization dataset from Sina Weibo containing 2.4 million training pairs.
- Taobao.com Production Title Summarization (Fine-tuning): A public dataset of human-generated product title summaries.
- JD.com Review Summarization (Fine-tuning): An in-house, human-labeled dataset from JD.com consisting of real customer reviews and their corresponding highlights.
Key Insights and Ideas
- Relevance Over Volume: The authors demonstrate that pre-training on a smaller amount of relevant data (summarization-specific) is more powerful than using massive general-purpose models. DSGPT-large (24 layers) is significantly smaller than GPT-3 but performs better on these specific tasks.
- Implicit Knowledge Wins: Unlike previous models that needed to be "fed" brand names and categories explicitly, DSGPT implicitly learns this background knowledge during the pre-training and fine-tuning phases. This makes the system much easier to maintain.
- Decoder-only Superiority: The paper argues that for text generation, the decoder-only structure is more "extensible" than the traditional encoder-decoder (Seq2Seq) models, as it handles the transition from input to output more fluidly.
Key Results
The model was evaluated using ROUGE scores (a standard metric for summarization quality).
- Title Summarization: On the Taobao dataset, DSGPT-large achieved a Rouge-L of 77.46, outperforming the previous SOTA (MS-Pointer_bi at 75.45), despite the latter using explicit background knowledge.
- Review Summarization: On JD.com’s internal data, DSGPT (Rouge-L 76.73) significantly outperformed standard Transformer models (Rouge-L 74.50).
- Length Control: Their "automatic length control" method improved the Rouge-L score on reviews from 76.73 to 79.59, proving that how you generate the text is just as important as the model itself.
- Qualitative Success: Case studies showed that DSGPT was better at retaining critical attributes (like "girls" vs "straight" for clothing) compared to baseline models, which often deleted essential descriptors.
Limitations and Future Work
While the paper focuses on Chinese e-commerce, the authors note that the framework is designed to be "simply extended to other domains." However, the paper does not explore how this model performs in low-resource languages where a dataset like LCSTS might not exist.
Future work could involve expanding DSGPT to a wider variety of e-commerce tasks, such as automated product description generation or multi-modal summarization (incorporating image data).
Summary updated