Introduction
Information Extraction (IE) is the process of transforming unstructured text into structured data, a task of critical importance in the biomedical field where research findings are buried in millions of scientific abstracts. Traditional sequence modeling techniques, such as standard Hidden Markov Models (HMMs), often treat sentences as flat strings of tokens. However, human language is inherently hierarchical, governed by grammatical structures that signal relationships between entities. This paper introduces Hierarchical Hidden Markov Models (HHMMs) to bridge the gap between linguistic structure and statistical sequence modeling, specifically for extracting complex biological interactions.
Hierarchical Architecture
The core innovation of this approach is a two-level modeling hierarchy that respects phrase boundaries. Rather than modeling a sentence as a single sequence of words, the HHMM operates at two distinct scales:
- The Phrase Level (Coarse): The top level represents the sentence as a sequence of phrase segments, such as Noun Phrases (NP) or Verb Phrases (VP). The system utilizes a shallow parser (the Sundance system) to pre-process text into these functional chunks.
- The Word Level (Fine): Nested within each phrase state is a secondary HMM that generates the specific words and their corresponding Part-of-Speech (POS) tags.
By using "typed" states—where a Noun Phrase state in the model can only emit a Noun Phrase from the text—the model is effectively constrained by grammar. This prevents the system from making ungrammatical predictions and ensures that entities like protein names are correctly identified as the heads of noun phrases.
Model States and Context Features
To handle the vast amount of irrelevant text in scientific abstracts, the researchers implemented a "null model" alongside extraction states. This allows the system to distinguish between sentences that describe a target relationship and those that merely mention entities in passing.
Furthermore, the authors introduced Context HHMMs (CHHMMs) to address the challenge of out-of-vocabulary words. Instead of evaluating words in isolation, the model examines a sliding window of three words and their POS tags. This grammatical context allows the model to make accurate predictions even when encountering previously unseen protein or chemical names, as it can infer the role of a word based on its surrounding linguistic environment.
Experimental Evaluation and Results
The HHMM was evaluated across three challenging biomedical relation-extraction tasks using MEDLINE abstracts:
- Subcellular-localization: Mapping proteins to their cellular locations.
- Disorder-association: Linking genes to specific diseases.
- Protein-interaction: Identifying physical interactions between protein pairs.
In the subcellular-localization and protein-interaction domains, the HHMM (particularly the context-aware variant) significantly outperformed flat baseline models, including Token HMMs and POS HMMs. The hierarchical models demonstrated superior precision across nearly all recall levels and achieved higher total recall, finding correct instances that flat models missed.
Interestingly, results for the disorder-association task were more competitive across all model types. This suggests that the linguistic signals for gene-disorder relations may be less dependent on complex phrase structures than other biological interactions, allowing simpler models to perform adequately.
Conclusion
By incorporating grammatical structure as a scaffold for statistical learning, Hierarchical Hidden Markov Models provide a more robust framework for information extraction in technical domains. While the system's performance is currently linked to the quality of initial shallow parsing, the move toward hierarchical modeling represents a significant step forward in capturing the long-distance dependencies and structural nuances of scientific language.
Summary updated