Abstract
Probing Pre-trained Language Models (PLMs) using prompts has indirectly implied that language models (LMs) can be treated as knowledge bases. To this end, this phenomena has been effective especially when these LMs are fine-tuned towards not just data of a specific domain, but also to the style or linguistic pattern of the prompts themselves. We observe that, satisfying a particular linguistic pattern in prompts is an unsustainable constraint that unnecessarily lengthens the probing task, especially because, they are often manually designed and the range of possible prompt template patterns can vary depending on the prompting objective and domain. We therefore explore an idea of using a position-attention mechanism to capture positional information of each word in a prompt relative to the mask to be filled, hence avoiding the need to re-construct prompts when the prompts linguistic pattern changes. Using our approach, we demonstrate the ability of eliciting answers to rare prompt templates (in a case study on health outcome generation) such as Postfix and Mixed patterns whose missing information is respectively at the start and in multiple random places of the prompt. More so, using various biomedical PLMs, our approach consistently outperforms a baseline in which the default mask language model (MLM) representation is used to predict masked tokens.
Introduction
The paradigm of treating Large Language Models (LLMs) as "Knowledge Bases" (LM-as-KB) relies on prompting—querying a model with a sentence containing a [MASK] token to elicit factual information. However, standard prompting is notoriously brittle. Models often become over-reliant on the specific linguistic structure of a prompt (e.g., always expecting the mask at the end) rather than the relational knowledge they contain. This "linguistic pattern constraint" makes manual prompt engineering time-consuming and difficult to scale across different domains.
Methodology: Position-based Prompting (PBP)
To overcome the limitations of structural dependence, we introduce Position-based Prompting (PBP). This method decouples factual recall from sentence structure by centering the model's attention on the gap it needs to fill.
Relative Position Conditioning
Instead of absolute positions, PBP utilizes relative positions centered on the [MASK] token (assigned an ID of 0). This allows the model to learn that context words are important based on their distance from the missing information, regardless of whether the mask appears at the start, middle, or end of the prompt.
Architecture and Custom Masking
The PBP framework adds a specialized position-attention layer on top of the PLM and incorporates query type embeddings to signal the prompt style. Furthermore, we employ custom masking during training. By specifically masking health outcome entities rather than random words, we force the model to prioritize medical facts. As shown in the training analysis, this custom approach allows the model to reach target perplexity significantly faster than standard random masking.
Experimental Results
We evaluated PBP using biomedical models like SciBERT, BioBERT, and UmlsBERT on clinical datasets including EBM-NLP and EBM-COMET.
Performance and Robustness
PBP consistently outperformed vanilla PLM baselines, achieving a 6.49% improvement in F1 score on the EBM-NLP dataset. SciBERT emerged as the strongest model for health outcome recall. Notably, PBP proved robust across "rare" prompt styles:
- Prefix prompts (mask at the end) remained the easiest, with up to 69.23% Exact Match accuracy.
- Postfix and Mixed prompts, which typically baffle standard models, became viable styles under the PBP framework.
Discussion and Limitations
While PBP significantly improves structural flexibility, challenges remain in the "long tail" of medical data. The model still struggles to recall infrequent outcomes that appeared rarely in the training set. Additionally, very long "Mixed" prompts (38+ tokens) continue to pose a challenge for accuracy. Future work will focus on scaling this position-aware mechanism to larger generative models and exploring its utility in other specialized scientific domains.
Conclusion
Position-based Prompting offers a more sustainable path for knowledge extraction from PLMs. By centering the coordinate system on the mask, we move away from rigid manual templates toward a more flexible, context-aware method of interrogating language models for domain-specific facts.
Summary updated