TL;DR
SALP-CG is a new large language model (LLM) pipeline designed to automatically classify and grade the privacy risks of online medical consultation data. By combining few-shot prompting, JSON Schema constraints, and deterministic rules, it achieves high accuracy (up to 0.900 Micro-F1) in identifying sensitive health information according to national security standards.
What Problem Does This Paper Address?
The digital transformation of healthcare has moved many doctor-patient interactions from the clinic to online consultation platforms. These "conversational health data" logs are a goldmine for research but a minefield for privacy. Unlike structured Electronic Medical Records (EMRs), chat logs are messy, informal, and filled with Protected Health Information (PHI).
Current privacy protection methods often rely on simple Named Entity Recognition (NER) to redact names or dates. However, modern data governance requires more than just "finding" entities; it requires classifying them into categories (e.g., symptoms vs. medications) and grading their sensitivity level (e.g., a common cold vs. an HIV diagnosis). Existing automated tools lack a unified standard for this grading, and traditional rule-based systems struggle with the nuanced context of human conversation—for example, distinguishing between a patient asking about a rare disease and a patient having one.
Methods
The authors introduce SALP-CG (Standard-Aligned LLM Pipeline for Classifying and Grading), a backend-agnostic framework that can be used with various LLMs (like GPT-4, Llama, or DeepSeek). The pipeline uses a three-layered approach:
- Few-Shot Guidance: The model is provided with 10 high-quality, manually labeled examples to help it understand the nuances of medical context and edge cases like negation (e.g., "patient does not have fever").
- JSON Schema Constrained Decoding: To ensure the LLM doesn't "hallucinate" or provide unstructured text, the authors use a JSON Schema. This forces the model to output data in a specific format and restricts its category choices to a predefined set, ensuring near-zero format errors.
- Deterministic High-Risk Rules: Recognizing that LLMs can still make mistakes on high-stakes data, the authors added a rule layer. For instance, specific regex patterns for high-risk lab results (like HIV or certain HPV genotypes) can override the LLM's output to ensure they are always graded at the highest sensitivity level (Level 5).
The grading logic is strictly aligned with the Chinese national standard GB/T 39725-2020, which categorizes health data into six types and five sensitivity levels.
Datasets
The study utilized the MedDialog-CN-2020 dataset, a massive collection of Chinese medical dialogues.
- Benchmark Set: The researchers sampled 1,000 data entries and had them manually labeled by a multidisciplinary team (experts in law, medicine, and computer science) to create a "gold standard" for evaluation.
- Synthetic Augmentation: Because many original records were already de-identified, the authors used the Python
Fakerlibrary to re-insert synthetic names, allowing them to test the pipeline's ability to catch direct identifiers.
Key Insights and Ideas
- The "Mosaic" Risk: The study found that while Level 4 and 5 data (highly sensitive) are relatively rare, Level 2 and 3 data (like dates, ages, and symptoms) dominate the landscape. While these seem innocuous individually, they can be combined to "re-identify" patients—a phenomenon the authors highlight as a major risk for data governance.
- Few-Shot is King: Ablation studies revealed that "few-shot guidance" was the single most important factor for performance. Removing it caused the largest drop in entity count accuracy and overall F1 scores.
- Contextual Intelligence: LLMs proved superior to traditional NER because they could handle "long-distance context." They could correctly grade a disease mentioned early in a chat based on a ruling-out statement made much later in the conversation.
- Schema as a Guardrail: Using JSON Schema constraints effectively "normalized" the model's behavior, leading to nearly 100% compliance with the required data categories across different model architectures.
Key Results
The pipeline was tested across 10 different LLMs. The highlights include:
- Top Performance: The strongest models (GPT-5-mini and GPT-5) achieved a Micro-F1 score of 0.900 for predicting the maximum sensitivity level of a record.
- Category Compliance: The Mean Category-Compatibility Rate (MCCR) was consistently high (≥ 0.984), proving the effectiveness of the JSON Schema constraints.
- Sensitivity Accuracy: GPT-4o-mini led the pack in sensitivity-level accuracy (MSGR) with a score of 0.995.
- Efficiency: The pipeline proved to be "backend-agnostic," meaning it worked reliably whether it was powered by OpenAI, Groq, or DeepSeek models.
Limitations and Future Work
- Language and Platform: The current benchmark is focused solely on Chinese medical dialogues. Future work is needed to see how well this generalizes to other languages and different types of medical platforms.
- Re-identification Nuance: While the tool identifies sensitive data, it doesn't yet quantify the exact risk of re-identification when multiple low-level data points are combined.
- Local Deployment: For maximum privacy, the authors suggest that future iterations should focus on local deployment of open-source LLMs within hospital intranets to prevent data from ever leaving the secure environment.
Summary updated