TL;DR
This paper introduces Text2Topic, a novel multi-label text classification system designed for efficient topic detection in user-generated content. It achieves high performance, even for unseen topics (zero-shot), by employing a bi-encoder transformer architecture and a smart dataset construction strategy, making it a practical and scalable solution for industry applications.
What Problem Does This Paper Address?
In the digital age, platforms like Booking.com are inundated with massive amounts of user-generated text (reviews, forum posts). Extracting structured insights from this data is crucial for understanding customer behavior and improving services. Multi-label text classification, the task of assigning multiple relevant topics to a piece of text, is a key challenge here. Existing methods often struggle with scalability to a large number of topics, efficient inference, and the ability to handle new, unseen topics without retraining. This paper tackles the need for a system that is accurate, scalable, cost-effective, and capable of zero-shot topic detection in a real-world, high-throughput environment.
Methods
The core of Text2Topic is a bi-encoder transformer architecture, inspired by Sentence-BERT. Unlike traditional cross-encoders that process text and topic together, the bi-encoder processes them separately. This allows for pre-calculating topic embeddings, significantly speeding up inference. The architecture uses concatenation, subtraction, and multiplication of text and topic embeddings, fed into feedforward layers.
Key methodological contributions include:
- Bi-Encoder Architecture: This is the central innovation, enabling efficient inference and zero-shot capabilities.
- Smart Dataset Construction: To build a comprehensive dataset, the authors employed optimized smart sampling and partial labeling techniques. This involved defining topics with natural language descriptions and using model-based sampling to identify relevant texts, reducing annotation costs and effort.
- Zero-Shot Capability: The bi-encoder architecture naturally supports zero-shot predictions, meaning it can classify texts into topics it hasn't seen during training, provided the topic has a descriptive definition.
- Production-Scale Deployment: The paper details the practical considerations for deploying the model on a real-world stream processing platform, focusing on high throughput and low latency.
Datasets
The paper describes a newly constructed dataset for Booking.com's use cases.
- Training/Evaluation Data: Approximately 1.6 million text-topic pairs were collected from user-generated content (reviews, forum posts) across three main data sources. This dataset defines 239 valuable topics, each with a name and description. Around 200K of these annotations are positive. The data is multilingual, including English, German, French, and Russian.
- Benchmark Comparison: The system is evaluated against baselines like MUSE (Multilingual Universal Sentence Encoder) and GPT-3.5.
Key Insights and Ideas
- Bi-encoder's Efficiency is Key: The bi-encoder architecture's ability to pre-compute topic embeddings is a game-changer for production-scale inference, drastically reducing latency compared to cross-encoders.
- Zero-Shot is Achievable and Valuable: The system demonstrates effective zero-shot capabilities, allowing it to adapt to new topics without costly retraining, a significant advantage for dynamic domains.
- Smart Data Annotation Pays Off: The authors show that intelligent sampling and partial labeling strategies can yield high-quality datasets at a lower cost, making large-scale annotation feasible.
- Domain-Specific Descriptions Matter: Defining topics with natural language descriptions, refined with tools like LIME, significantly aids both model training and zero-shot performance.
- Single Model Outperforms Multiple: Training a single, comprehensive model on data from all sources proved more effective than training separate models for each data source, highlighting the benefits of generalization.
Key Results
- High Accuracy: The deployed Text2Topic model achieved a micro mAP of 92.9% and a macro mAP of 75.8% on the test set, outperforming state-of-the-art baselines.
- Efficient Inference: The system supports production-scale batch inference with high throughput, achieving 8000 texts per minute at a significantly lower cost than LLMs like GPT-3.5.
- Zero-Shot Performance: Text2Topic's zero-shot capabilities were found to be better than MUSE for topics within the travel domain.
- LLM Comparison: Text2Topic outperformed GPT-3.5 (8-shot and 38-shot prompts) on a set of 24 topics, offering a more feasible, cost-effective, and controllable solution for industry use.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while the bi-encoder concatenation method is better than simple cosine similarity, embedding subtraction and multiplication are both necessary for optimal performance.
- The performance of zero-shot capabilities can depend on how well-defined the unseen topic description is.
- While Text2Topic outperforms GPT-3.5 in many aspects, LLMs might still be better for certain niche or highly nuanced topics, though at a much higher cost and complexity.
- Future Work:
- Explore parameter-efficient fine-tuning techniques (e.g., LoRA, p-tuning) on open-source LLMs to potentially improve performance.
- Investigate how to better balance model specialization and generalization power for zero-shot scenarios.
Summary updated