Boolean-aware Attention for Dense Retrieval

Abstract

Standard transformer-based dense retrieval models often struggle to interpret Boolean logic, such as "AND", "OR", and "NOT", frequently retrieving documents that contain terms a user explicitly intended to exclude. We introduce BoolAttn, a modular, plug-and-play attention mechanism that detects Boolean operators and dynamically adjusts token attention weights. By integrating BoolAttn into BERT, we demonstrate significant improvements in dense retrieval performance on complex, logic-heavy queries.

Introduction

When users perform highly specific searches, they often rely on Boolean logic—either explicitly (e.g., "Movies set in Vietnam but not about war") or implicitly (e.g., "Animals found only in Brazil and Mexico").

Traditional lexical matchers like BM25 often fail in these scenarios, particularly with exclusions, because they prioritize word overlaps and may retrieve documents containing the exact terms the user wished to avoid. Similarly, standard dense retrieval models based on transformers like BERT lack an explicit mechanism to interpret Boolean cues and their contextual scope. They treat logical operators as regular tokens, leading to suboptimal retrieval when strict logical semantics are required.

Methodology

The proposed Boolean-aware attention (BoolAttn) is a modular enhancement designed to integrate directly into standard self-attention layers. It consists of four primary components:

1. Cue Predictor: Identifies the position of Boolean cues (e.g., "not", "excluding", "and") within the input sequence.
2. Scope Predictor: Determines the span of tokens affected by a detected operator using a Conv1D layer for local dependencies and a FiLM (Feature-wise Linear Modulation) layer for operator adaptation.
3. Bias Predictor: Computes an attention bias based on the distance from the cue, utilizing a Gaussian kernel to weight positions under the hypothesis that tokens closer to the cue are more heavily influenced.
4. Operator-Gating Mechanism: Dynamically activates specific "Boolean experts" based on the detected operator.

Expand

Source, p. 2

The calculated Boolean bias is added to the raw attention scores before the softmax step. For "AND" and "OR", attention is amplified; for "NOT", the attention scores of scoped tokens are explicitly downweighted to suppress their influence on the query representation.

Experimental Setup

The model was evaluated using several key datasets:

• Pretraining Dataset: A GPT-generated dataset used to pretrain the Cue Predictor and Gate modules on auxiliary tasks like cue prediction and expert classification.
• QUEST: A retrieval dataset of entity-seeking queries with implicit set operations, testing the model's ability to understand logical relationships among entities.
• BoolQuestions: A dataset derived from MS MARCO focusing on explicit AND, OR, and NOT operations.

Results and Discussion

Performance on Complex Queries

Integrating BoolAttn into BERT-base significantly improved performance across logic-heavy tasks. On the QUEST dataset, Average Recall@100 improved from 0.443 to 0.483 for BERT-base and from 0.467 to 0.495 for BERT-large.

Expand

Source, p. 4

On the BoolQuestions dataset, Bool-BERT-base achieved a Mean Reciprocal Rank (MRR@10) of 0.480 for AND queries, 0.586 for OR queries, and 0.165 for NOT queries, outperforming standard BERT-base and prior baselines like distillbert-dot-v5.

Key Insights

• The "Pink Elephant" Paradox: Language models often struggle to ignore negated terms because they must represent a concept to negate it. BoolAttn solves this by explicitly subtracting attention bias from negated tokens, suppressing them from the final query representation.
• Operator-Specific Scopes: The model successfully learns that different operators have different structural rules; for instance, "NOT" is typically unidirectional, while "AND" and "OR" are bidirectional.
• Dynamic Positive Sampling: Sampling different positive documents at each training step for multi-answer queries was found to improve model robustness significantly compared to fixed positive sampling.

Conclusion

BoolAttn successfully teaches BERT to respect logical constraints through a plug-and-play module. However, challenges remain, including a roughly 10% increase in parameters and potential error propagation if the Cue Predictor misses implicit negations. Future work will focus on mitigating computational overhead through parameter-sharing and improving the gating mechanism to avoid performance bottlenecks.

@misc{mai2025boolean,
  author = {Quan Mai and Susan Gauch and Douglas Adams},
  title = {{Boolean-aware Attention for Dense Retrieval}},
  year = {2025},
  eprint = {2503.01753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.01753},
}