Introduction
The DEFT 2019 (Défi Fouille de Texte) challenge focused on the complexities of the French medical domain. Qwant Research participated in two primary tasks: matching clinical cases to their corresponding discussions (Task 2) and extracting structured demographic and clinical information from medical narratives (Task 3). The research highlights a critical lesson in specialized NLP: while sophisticated neural architectures are powerful, success on small, domain-specific datasets often hinges on robust preprocessing and clever data augmentation.
Task 2: Semantic Matching
The objective of Task 2 was to pair a clinical case with its correct discussion from a pool of candidates. The team approached this as a retrieval problem using a Query Likelihood Model. By treating each discussion as a language model, they calculated the perplexity of a clinical case given that model; lower perplexity indicated a higher likelihood of a match.
A significant innovation in their pipeline was a "de-abbreviation" lexicon. Medical texts are dense with non-standard abbreviations (e.g., IVG), which can be ambiguous. By implementing a lexicon of 227 terms—resulting in over 9,000 substitutions—the team significantly reduced noise. Furthermore, they employed an exclusive matching strategy, enforcing a one-to-one mapping between cases and discussions to improve overall system precision.
Task 3: Information Extraction
Task 3 required the extraction of five key variables: age, gender, reason for admission, and clinical outcome. The researchers explored two distinct paths:
- Neural Architecture: They implemented a hybrid Bi-LSTM-CNN-CRF model, a state-of-the-art structure for Named Entity Recognition (NER). This model uses a Convolutional Neural Network (CNN) to capture character-level features and a Bidirectional Long Short-Term Memory (Bi-LSTM) network for sequence modeling, followed by a Conditional Random Field (CRF) layer to optimize label sequences.
- Linguistic Rules: For variables like age, they used dependency parsing to identify grammatical links between numerical values and temporal markers (e.g., "years").
To combat the scarcity of training data (only 290 cases), the team used a Context-Free Grammar (CFG) to generate 2,000 synthetic clinical case introductions. This synthetic data allowed the neural models to learn the standard linguistic patterns of patient descriptions more effectively.
Key Results and Insights
The experimental results yielded several surprising insights into the behavior of models on specialized corpora.
The "Bag of Words" Surprise
In Task 2, simpler 1-gram (unigram) models outperformed more complex 2-gram or 3-gram models on the evaluation set. This suggests that higher-order models overfit the limited training data, whereas unigram models generalized better to the unseen test set.
Normalization and Performance
The impact of data cleaning cannot be overstated. Automating the expansion of abbreviations provided a massive performance boost, contributing nearly 19 points of improvement in precision. This underscores that in specialized domains, data quality often outweighs algorithmic complexity.
Final Performance
The team achieved a best precision of 84.11% for document matching. For information extraction, performance was exceptionally high for demographic data (F1-scores of 0.93 for age and 0.95 for gender) but lower for clinical outcomes (0.59 F1), which are often implied rather than explicitly stated.
Conclusion
The Qwant Research contribution to DEFT 2019 demonstrates that effective medical NLP requires a balance of modern deep learning and traditional linguistic engineering. While neural models like Bi-LSTM-CNN-CRF are excellent for entity extraction, they require support from synthetic data and rigorous preprocessing—particularly de-abbreviation—to overcome the challenges of small, highly specialized datasets. Future work will focus on refining the extraction of complex clinical outcomes and exploring bidirectional matching strategies.
Summary updated