Sure, here's a blog-style summary of the paper "Automatic Construction of Enterprise Knowledge Base":
TL;DR
This paper presents a system for automatically building knowledge bases from enterprise documents, addressing the challenge of data distribution shifts and the need for accuracy in enterprise settings. By combining state-of-the-art deep learning with classical NLP techniques, the system effectively extracts and organizes information, serving as a valuable tool for knowledge discovery within organizations.
What Problem Does This Paper Address?
Enterprises are awash in vast amounts of internal documents – emails, reports, presentations, and more. While public knowledge bases like Wikipedia have revolutionized web search, creating similar, tailored knowledge bases for individual companies from their private data has been a significant hurdle. The core problem lies in the unique characteristics of enterprise data: it often exhibits a "data distributional shift" compared to public datasets, meaning models trained on public data struggle to perform well. Furthermore, enterprise knowledge bases demand high accuracy and must be fully automated without extensive human customization or reliance on pre-existing structured data. This paper tackles the challenge of building such a system that can automatically mine, organize, and present knowledge from a company's internal documents.
Methods
The system employs a multi-stage approach, combining deep learning and traditional NLP techniques.
- Named Entity Recognition (NER): The system uses a Transformer-based model, fine-tuned in multiple stages. It starts with a large, automatically annotated dataset from Wikipedia to help generalize to new domains. This is followed by training on higher-quality datasets like OntoNotes and then a custom dataset derived from internal documents. Techniques like data augmentation (e.g., random entity replacement) and focal loss are used to improve performance on imbalanced datasets. Viterbi decoding is employed to ensure valid entity sequences. A key advancement is the use of a larger, more diverse pre-trained model (UniLMv2-large) which significantly boosts performance.
- Definition Mining: A deep learning model classifies sentences into categories like "Sufficient," "Informational," and "Referential" definitions. This pipeline involves splitting documents into sentences, classifying them, and then extracting topic-definition pairs. To address the distributional shift, they create both a public dataset (from Bing search results) and an enterprise dataset (from internal, compliance-approved documents).
- Topic Ranker: To manage the massive number of potential topics identified by NER, a two-stage ranking system is used. First, topics are ranked by their frequency. Then, a binary classifier, trained on features like NER frequency and document frequency, further refines the list to remove noisy topics.
- Topic Card Builder: This component aggregates information from other modules to create comprehensive "topic cards." It computes relatedness between topics, documents, and users using dense embeddings derived from topic-document matrices (using a randomized SVD algorithm) and user document embeddings. This enables features like expert finding and topic conflation.
Datasets
- Public Datasets: Wikipedia, CoNLL03, OntoNotes 5.0, DEFT corpus, and Bing search query logs were used for training and evaluation.
- Enterprise Datasets: A custom "enterprise dataset" was created from internal Microsoft documents that have undergone compliance review. This dataset is crucial for evaluating and training models on domain-specific data.
Key Insights and Ideas
- Distributional Shift is Real and Significant: The paper strongly emphasizes that models trained on public data perform poorly on enterprise data. This necessitates domain-specific training and evaluation.
- Multistage Training is Crucial: A single-stage training approach is insufficient. Iterative training on progressively more relevant data (Wikipedia -> OntoNotes -> Enterprise) significantly improves performance.
- Leveraging Enterprise Data is Key: While public data helps with generalization, the real performance gains come from training and evaluating on actual enterprise documents.
- Beyond Basic NER: Enterprise knowledge bases require more than just identifying common entities. The inclusion of "project," "field of study," and "creative work" as entity types is a practical adaptation to enterprise needs.
- User-Centric Design Matters: The system's ability to link users to topics and related people ("expert finding") is a valuable feature for enterprise productivity.
Key Results
- The best model achieved an F1 score of 71.1% on the internal enterprise test set, a significant improvement over the baseline (56.0%).
- Multistage training improved F1 by 5.4% over the baseline.
- Using a larger pre-trained model (UniLMv2-large) provided a 5.7% F1 lift.
- Knowledge distillation achieved approximately 6x speedup for inference, with further optimizations leading to an additional ~14x speedup.
- The definition classification model achieved an F1 of 0.75 on the enterprise dataset.
- Human evaluation showed an 85% good rate for generated topic cards.
Limitations and Future Work
- Compliance Restrictions: The use of enterprise data is subject to strict compliance rules, which can limit data access and model training environments.
- Data Distributional Shift: While addressed, this remains a persistent challenge, and further research into domain adaptation techniques is implied.
- Scalability: The paper mentions the challenge of handling massive matrices for embedding computation, and their solution with randomized SVD is a step towards scalability.
- Future Work: The paper suggests that further improvements can be made by exploring more advanced NLP techniques and potentially incorporating more diverse data sources for pre-training. The system is already in production as part of Microsoft 365 (Microsoft Viva Topics), indicating ongoing development and refinement.
Summary updated