DeepSuggest: Using Neural Networks to Suggest Related Keywords for a Comprehensive Search of Clinical Notes

Soheil Moosavinasab, Emre Sezgın, Huan Sun, Jeffrey Hoffman, Yungui Huang, Simon LinACI 2021 · DOI

Abstract

Objective: A large amount of clinical data are stored in clinical notes that frequently contain spelling variations, typos, local practice-generated acronyms, synonyms, and informal words. Instead of relying on established but infrequently updated ontologies with keywords limited to formal language, we developed an artificial intelligence (AI) assistant (named “DeepSuggest”) that interactively offers suggestions to expand or pivot queries to help overcome these challenges. Methods: We applied an unsupervised neural network (Word2Vec) to the clinical notes to build keyword contextual similarity matrix. With a user's input query, DeepSuggest generates a list of relevant keywords, including word variations (e.g., formal or informal forms, synonyms, abbreviations, and misspellings) and other relevant words (e.g., related diagnosis, medications, and procedures). Human intelligence is then used to further refine or pivot their query. Results: DeepSuggest learns the semantic and linguistic relationships between the words from a large collection of local notes. Although DeepSuggest is only able to recall 0.54 of Systematized Nomenclature of Medicine (SNOMED) synonyms on average among the top 60 suggested terms, it covers the semantic relationship in our corpus for a larger number of raw concepts (6.3 million) than SNOMED ontology (24,921) and is able to retrieve terms that are not stored in existing ontologies. The precision for the top 60 suggested words averages at 0.72. Usability test resulted that DeepSuggest is able to achieve almost twice the recall on clinical notes compared with Epic (average of 5.6 notes retrieved by DeepSuggest compared with 2.6 by Epic). Conclusion: DeepSuggest showed the ability to improve retrieval of relevant clinical notes when implemented on a local corpus by suggesting spelling variations, acronyms, and semantically related words. It is a promising tool in helping users to achieve a higher recall rate for clinical note searches and thus boosting productivity in clinical practice and research.

Introduction: Bridging the Gap Between Formal Ontologies and Clinical Reality

Clinical notes are a treasure trove of patient data, but they are notoriously difficult to search. While researchers and clinicians need to find specific cohorts or outcomes, the language used in these notes is often informal, riddled with typos, and heavy on local shorthand (e.g., "T&A" for tonsillectomy and adenoidectomy). Traditional search engines in Electronic Health Record (EHR) systems typically rely on formal ontologies such as SNOMED. These ontologies are standardized but static, often failing to account for the linguistic "noise" of real-world clinical practice. This creates a vocabulary mismatch: formal search terms do not match the messy, abbreviated reality of clinical documentation.

Methods: The DeepSuggest Architecture

DeepSuggest is an interactive query expansion tool built on an unsupervised neural network. The system follows a two-stage process: offline training and online interactive querying.

The DeepSuggest system architecture, illustrating the offline training pipeline and the online interactive query process.
  1. Word Embedding: The system uses the Word2Vec (Continuous Bag-of-Words) algorithm to map words into a 400-dimensional vector space, where words in similar contexts are positioned closely.
  2. N-gram Processing: DeepSuggest handles sequences of 1 to 4 words (n-grams), using Pointwise Mutual Information (PMI) to identify meaningful phrases like "bilateral tube insertion."
  3. Fast Retrieval: To enable real-time interaction, the model is indexed using Annoy (Approximate Nearest Neighbors).
  4. Human-in-the-Loop: The tool presents a dashboard of suggestions for the user to refine or pivot their query.

Implementation and User Interface

The model was trained on a massive local corpus from Nationwide Children’s Hospital (NCH), encompassing over 66 million clinical notes (approx. 5.5 billion words). The resulting vocabulary consists of 6.3 million unique n-grams.

The DeepSuggest user interface, showing the search dashboard with contextual estimates, keyword suggestions, and sample notes.

The interface allows users to see "Contextual Estimates" and "Suggestions" (e.g., for "tonsillectomy," it might suggest "t&a" or "adenoidectomy"). This "semantic pivoting" helps users discover related concepts, such as social determinants of health (e.g., "bus tickets" suggested for "meal voucher"), that are absent from traditional medical ontologies.

Key Results and Performance

DeepSuggest significantly outperforms standard EHR search tools in recall. In usability tests with medical residents, the system retrieved an average of 5.6 relevant notes per search, compared to just 2.6 notes using the native Epic EHR search—a nearly 2x improvement.

While DeepSuggest recalled only 54% of formal SNOMED synonyms, it excelled at finding the informal and "noisy" variations (like "tonsilectomy") that actually appear in clinical notes. The precision for the top 60 suggested words averaged 0.72, indicating high clinical relevance.

Discussion and Future Directions

DeepSuggest demonstrates the power of using AI as a "thought partner" for researchers, helping them dynamically refine the scope of their studies based on actual data content. However, the current model uses Word2Vec, which is not context-aware. Future work may involve integrating context-sensitive models like BERT or ELMo to distinguish between different meanings of the same word (e.g., "cold" as a temperature vs. a virus). Additionally, combining AI-driven suggestions with formal ontologies could create a hybrid system offering both formal accuracy and real-world flexibility.

Summary updated

Abstract

Abstract Objective A large amount of clinical data are stored in clinical notes that frequently contain spelling variations, typos, local practice-generated acronyms, synonyms, and informal words. Instead of relying on established but infrequently updated ontologies with keywords limited to formal language, we developed an artificial intelligence (AI) assistant (named “DeepSuggest”) that interactively offers suggestions to expand or pivot queries to help overcome these challenges. Methods We applied an unsupervised neural network (Word2Vec) to the clinical notes to build keyword contextual similarity matrix. With a user's input query, DeepSuggest generates a list of relevant keywords, including word variations (e.g., formal or informal forms, synonyms, abbreviations, and misspellings) and other relevant words (e.g., related diagnosis, medications, and procedures). Human intelligence is then used to further refine or pivot their query. Results DeepSuggest learns the semantic and linguistic relationships between the words from a large collection of local notes. Although DeepSuggest is only able to recall 0.54 of Systematized Nomenclature of Medicine (SNOMED) synonyms on average among the top 60 suggested terms, it covers the semantic relationship in our corpus for a larger number of raw concepts (6.3 million) than SNOMED ontology (24,921) and is able to retrieve terms that are not stored in existing ontologies. The precision for the top 60 suggested words averages at 0.72. Usability test resulted that DeepSuggest is able to achieve almost twice the recall on clinical notes compared with Epic (average of 5.6 notes retrieved by DeepSuggest compared with 2.6 by Epic). Conclusion DeepSuggest showed the ability to improve retrieval of relevant clinical notes when implemented on a local corpus by suggesting spelling variations, acronyms, and semantically related words. It is a promising tool in helping users to achieve a higher recall rate for clinical note searches and thus boosting productivity in clinical practice and research. DeepSuggest can supplement established ontologies for query expansion.
BibTeX
@inproceedings{moosavinasab2021deepsuggest,
  author = {Soheil Moosavinasab and Emre Sezgın and Huan Sun and Jeffrey Hoffman and Yungui Huang and Simon Lin},
  title = {{DeepSuggest: Using Neural Networks to Suggest Related Keywords for a Comprehensive Search of Clinical Notes}},
  booktitle = {ACI},
  year = {2021},
  doi = {10.1055/s-0041-1729982},
}
Figure