Introduction
The 2019 DEFT (Défi Fouille de Textes) challenge focused on two critical tasks in clinical Natural Language Processing (NLP): indexing clinical documents with medical keywords and pairing clinical cases with their corresponding professional discussions. Clinical texts present unique challenges due to specialized terminology, abbreviations, and dense syntax. This paper explores various strategies, ranging from unsupervised string matching to deep learning, to address these tasks using the French-language DEFT 2019 corpus.
Task 1: Indexing Clinical Documents
The goal of Task 1 was to automatically assign relevant medical keywords from the MeSH (Medical Subject Headings) thesaurus to pairs of clinical cases and discussions. The authors tested several approaches:
- Pairing-based (Run 1): An endogenous method that assumed documents should share keywords with similar documents in the training set.
- Terminological Annotation (Run 4): A hybrid approach using the Whoosh search engine, Pointwise Mutual Information (PMI), and Latent Semantic Analysis (LSA) to map terms to MeSH concepts.
Despite the complexity of these methods, they were significantly outperformed by a simple baseline that checked for the presence of reference keywords. The sophisticated LSA/PMI model struggled to capture the formal hierarchy of MeSH effectively in this context.
Task 2: Pairing Cases and Discussions
Task 2 involved matching a patient's medical history (clinical case) with the correct professional commentary. The authors compared two distinct philosophies:
- Character-level Distributions (Run 1): An unsupervised method identifying "hapax" strings—unique sequences shared between a case and a discussion to link documents.
- Siamese Networks (Run 2 & 3): A deep learning approach using multi-layer perceptrons to map documents into a shared feature space, with similarity determined by Euclidean distance and global assignment solved via the Hungarian algorithm.
The results revealed a surprising outcome: the simple character-matching approach far exceeded the performance of the neural models.
Discussion and Key Insights
The most striking finding was the "Simpler is Better" result in the pairing task. While the Siamese networks plateaued quickly with a precision of roughly 11-13%, the unsupervised character similarity method achieved 61.7%. The authors suggest that clinical discussions often "echo" specific, unique identifiers or phrases from the case. Character-level matching captures these direct links, whereas semantic embeddings in neural networks may smooth over these crucial markers, especially when training data is limited.
In contrast, the indexing task proved more difficult than anticipated. The authors' hypotheses regarding pairing-based indexing and terminological annotation failed to beat the baseline, highlighting the difficulty of mapping raw clinical text to formal hierarchies like MeSH without a massive amount of training data.
Conclusion
While the results for Task 1 and the deep learning components of Task 2 were below expectations, the success of the character-matching approach provides a valuable lesson in clinical NLP. For specific document-pairing tasks where unique identifiers are present, simple string-based methods can remain more robust than complex neural architectures, particularly in low-resource or highly specialized domains like French clinical medicine.
Summary updated