TL;DR
This paper tackles the challenge of extracting meaningful information from noisy customer-agent interactions in an Enterprise Resource Planning (ERP) context. They find that the language used in these interactions is surprisingly divergent from standard Romanian, even in a business setting, and propose a hybrid approach combining TF-IDF and word embeddings for effective content cleanup and extraction.
What Problem Does This Paper Address?
Companies collect vast amounts of data from customer-agent interactions (emails, chats, etc.) which, while valuable, are often "noisy." This noise comes from various sources: informal language, code-switching (mixing languages), abbreviations, and extraneous information like email headers and disclaimers. Standard Natural Language Processing (NLP) techniques struggle with this kind of data, leading to inaccurate analysis, classification, and response generation. The authors specifically focus on Romanian, a language with limited NLP resources, and aim to bridge the gap between the real-world language used in business communications and the standardized language found in dictionaries.
Methods
The authors employ a two-pronged approach:
- Lexical Analysis: They compare the vocabulary used in their customer-agent interaction corpus (questions and answers) against:
- A standard Romanian dictionary (DEX).
-
A large general-purpose corpus (Common Crawl). They use word embeddings (specifically CBOW with negative sampling) to analyze word usage and identify overlaps and differences.
-
Content Extraction: To clean the noisy data, they develop a hybrid method that combines:
- TF-IDF (Term Frequency-Inverse Document Frequency): A standard technique for identifying important words in a document.
- Word Embeddings: Used to create sentence representations by averaging the embeddings of words within a sentence. This hybrid approach is used to train a binary classifier to identify and remove irrelevant sentences.
Datasets
- Question-Answering (Q/A) Corpus: A proprietary dataset of email exchanges from a Romanian company's ERP environment. This corpus was anonymized and contains approximately 18.7 million tokens.
- Romanian Explicative Dictionary (DEX): Used as a model of standardized Romanian vocabulary.
- Common Crawl: A large corpus of Romanian texts used for comparison with general language usage.
- Annotated Dataset: For content extraction, the authors manually annotated approximately 2000 requests (200,000 words) to create a benchmark for training their classifier.
Key Insights and Ideas
- Surprising Divergence: The language used in business customer-agent interactions is significantly different from standardized Romanian, even more so than general web text (Common Crawl). This highlights the need for domain-specific adaptation.
- Code-Switching and Non-Standardisms: Business communication, contrary to expectations, exhibits a high prevalence of code-switching (Romanian-English), abbreviations, and adapted foreign words, likely to speed up communication.
- Lexical Richness in Customer Queries: Customers tend to use a richer vocabulary in their questions compared to agents' replies, suggesting a higher probability of encountering new or less common words in customer queries.
- Hybrid Approach Effectiveness: Combining TF-IDF and word embeddings for sentence classification proves more robust than using either method alone, especially for handling out-of-domain content and preserving relevant information.
Key Results
- Low Dictionary Overlap: The vocabulary in the Q/A corpus shows at most a 60% overlap with the standardized Romanian dictionary.
- Content Extraction Performance: The hybrid classifier achieved a cross-validation accuracy of 0.89 and an F1 score of 0.74 for sentence removal. While TF-IDF alone performed well (0.89 accuracy, 0.74 F1), the hybrid approach showed better generalization on unseen data.
- Improved Robustness: The combined TF-IDF and embedding model demonstrated a "less restrictive behavior," preserving more relevant content and generalizing better across different topics compared to TF-IDF alone.
Limitations and Future Work
- Limited Romanian NLP Resources: The authors acknowledge the scarcity of NLP resources for Romanian, which motivates their work and their decision to release parts of their data.
- Classifier Rigidity: The initial TF-IDF classifier was found to be too rigid and biased towards the training data, leading to the removal of some valid sentences. The hybrid approach aimed to mitigate this.
- Human Evaluation: While promising results were obtained, human evaluation is underway to assess the quality of the extracted sentences on larger subsamples.
- Future Work: The authors hope their work will increase the availability of digital resources for Romanian NLP and encourage further research in this area. They also suggest that the extracted relevant content can be used for downstream tasks like classification, retrieval, and automatic reply generation.
Summary updated