TL;DR
This paper introduces BACCHANT, a machine learning system for normalizing bioscience text by classifying punctuation. They found that normalizing text within named entities (like gene or protein names) is significantly harder, leading to the development of BACCHANT-N, a specialized system that outperforms general normalization when applied to these specific entities. This suggests that integrating named entity recognition with tokenization could be a more effective approach for bioscience text.
What Problem Does This Paper Address?
Tokenization, the process of breaking down text into meaningful units (tokens), is a fundamental step in many natural language processing tasks. However, bioscience literature presents unique challenges: a vast and evolving vocabulary, technical jargon, and non-standard orthography. This makes it difficult to consistently identify token boundaries. For instance, terms like "NF-kappaB" can appear with spaces, hyphens, or no separators at all, leading to inconsistent tokenization. This inconsistency can hinder downstream tasks like information retrieval, where a search for "NF-kappaB" might miss documents using a different variant. Normalization aims to address this by standardizing tokens, often by removing punctuation that doesn't signify a true boundary. The paper specifically investigates how the presence of named entities (like gene or protein names) affects the difficulty of this normalization task.
Methods
The core of the paper's methodology lies in framing tokenization and normalization as a classification problem. Specifically, they treat punctuation marks as candidates for either breaking a token or being removed for normalization.
- BACCHANT (Bioscience And Health Article Normalizing Tokenizer): This system uses a decision tree classifier. It learns to classify punctuation based on a set of features derived from the surrounding characters and the punctuation itself. Key features include:
- The punctuation mark itself.
- The character immediately to its left and right.
- The type of these surrounding characters (e.g., lowercase, uppercase, number, space).
- The context of the punctuation within the sentence or article.
- The classification outcome is either 'remove' (normalize by removing punctuation) or 'break' (use punctuation as a token boundary).
- BACCHANT-N: Recognizing that normalization within named entities might be different, they created a second system. BACCHANT-N is trained exclusively on data from within named entities extracted from the GENIA corpus. This system uses the same features and a decision tree classifier as BACCHANT.
- Feature Engineering: The authors experimented with various features and used information gain ratio to select the most informative ones for their decision tree.
Datasets
- MEDLINE Abstracts: The initial training and testing for BACCHANT were performed on a set of 67 MEDLINE abstracts, hand-tokenized by the authors. This dataset yielded approximately 17,253 punctuation instances.
- GENIA Corpus: For evaluating performance specifically within named entities, the authors used a subset of the GENIA corpus. This corpus consists of MEDLINE abstracts that have been hand-tagged for named entities. The test set for BACCHANT-N was derived from the punctuation found within these named entities, totaling 5,798 instances.
- Gold Standards: The authors provide links to download their hand-tokenized gold standards for both "inside named entities" and "all text" (inside and outside named entities).
Key Insights and Ideas
- Punctuation as a Classification Problem: The framing of punctuation classification as a machine learning task is a key methodological contribution, allowing for data-driven normalization.
- Named Entities are "Harder": The most significant insight is that normalizing text within named entities is demonstrably more challenging than normalizing general text. This is evidenced by the performance drop of BACCHANT when tested on named entity data compared to general text.
- Specialization Yields Improvement: Training a dedicated normalization system (BACCHANT-N) solely on named entity data leads to a statistically significant improvement in performance for normalizing named entities, even surpassing the performance of a system trained on a broader dataset.
- Pipelines vs. Integration: The findings strongly suggest that a standard pipelined approach (tokenization, then named entity recognition) might be suboptimal for bioscience text. Instead, integrating tokenization and named entity tagging processes could be more beneficial.
Key Results
- BACCHANT Performance: BACCHANT achieved an accuracy of 96.60% on general MEDLINE text, a statistically significant improvement over a simple baseline that breaks on all punctuation (92.73% accuracy).
- BACCHANT on Named Entities: When tested on named entity data from the GENIA corpus, BACCHANT's accuracy dropped to 90.0%.
- BACCHANT-N Performance: BACCHANT-N, trained exclusively on named entity data, achieved an accuracy of 96.5% when tested on named entity data. This represents a significant improvement over BACCHANT's 90.0% on the same data.
- Precision/Recall for 'Remove' Class: For the critical 'remove' class (punctuation to be normalized), BACCHANT achieved a precision of 0.832 and a recall of 0.688. BACCHANT-N improved recall to 0.789 while maintaining a similar precision of 0.833.
Limitations and Future Work
- Limited Context: BACCHANT currently only considers one character to either side of a punctuation mark. The authors suggest expanding this context window to potentially improve accuracy and learn more nuanced features.
- Need for Integrated Systems: The findings highlight the need for systems that can perform named entity tagging and normalization simultaneously, rather than in a sequential pipeline. The authors propose exploring hierarchical hidden Markov models as a potential avenue for such integrated systems.
- Domain Specificity: While BACCHANT-N performs better on named entities, it is not designed to handle text outside of named entities. This necessitates a system that can handle both aspects effectively.
Summary updated