CANDI: Contextual Alignment for Niche Domains Question Answering

Megha Chakraborty, Darssan L. Eswaramoorthi, Het Riteshkumar Shah, Madhur Thareja, Michelle A Ihetu, Harshul Raj Surana, Kaushik Roy, Amit ShetharXiv 2026 · arXiv

TL;DR

While Large Language Models (LLMs) excel at general knowledge, they often stumble in highly specialized, multi-stakeholder professional environments. This paper introduces CANDI-QA, a novel benchmark designed to evaluate "contextual alignment" within the niche domain of school behavioral health support, and proposes MTSS-Net, a neuro-symbolic baseline framework that demonstrates how structured, role-aware reasoning can significantly boost LLM performance where standard prompting fails.

What Problem Does This Paper Address?

When we talk about "AI alignment," we usually mean safety, factuality, or general human preferences. However, real-world enterprise and institutional settings require a different flavor of alignment: contextual alignment. This is a model's ability to tailor its responses based on the specific domain, the exact topic of conversation, and—crucially—the professional role and background knowledge of the user.

The authors focus on a highly complex, real-world use case: the Multi-Tiered System of Supports for Behavioral Health (MTSS-B) in schools.

Figure 2: MTSS (B)

As shown in Figure 2, MTSS-B involves a multi-disciplinary team (teachers, counselors, administrators, psychologists) operating across three distinct tiers of intervention (from school-wide prevention to intensive individual care). A generic LLM response to a behavioral health question might be factually correct but practically useless if it ignores whether the person asking is a school nurse or a principal, or whether the student in question requires Tier 1 or Tier 3 support. Traditional QA benchmarks completely miss this nuance.

Methods

To systematically evaluate and improve contextual alignment, the authors designed a dual-pronged approach:

1. Prompting Strategies

They tested ten different language models across three distinct prompting setups:

  • P1 (No-Context): Only the user query is provided.
  • P2 (Role-Aware): The system prompt includes the user's specific role and responsibilities.
  • P3 (Task-Aware): The system prompt includes a high-level objective of the AI assistant alongside the role description.

2. Evaluation Framework

Because different questions require different types of thinking, the evaluation is split by question type:

  • Factual Lookup (T1): Evaluated against expert "gold-standard" answers using traditional semantic and syntactic metrics (Cosine Similarity, BERTScore, MoverScore, BLEURT).
  • Applied Inference (T2): Since these lack a single correct answer, the authors used a reference-free Retrieval-Augmented Generation (RAG) evaluation framework (RAGAs), measuring context precision, response relevancy, faithfulness, and semantic similarity against a strong reference context generated by a Qwen model.

3. The Baseline: MTSS-Net

To move beyond raw prompting, the authors introduced MTSS-Net, a lightweight neuro-symbolic framework:

  • v1 (Structured Context): Encodes each QA instance into a structured JSON object capturing the task type, user role, intervention tier, and relevant policy references.
  • v2 (Role-Aware Reasoning & Attribution): Adds curated expert QA pairs, explicit role conditioning, and source citations to ground the model's reasoning.

Datasets

The paper introduces CANDI-QA, a proprietary dataset meticulously curated with human experts from the School Behavioral Health Academy (SBHA).

Figure 3: Features of CANDI-QA

As illustrated in Figure 3, the dataset consists of 315 total questions split into two categories:

  • T1: Information Assistance Questions (90 questions): Direct, factual queries regarding best practices and evidence-based literature.
  • T2: Applied Inference Questions (225 questions): Complex, multi-hop reasoning scenarios that are student- or situation-specific, requiring role-sensitive interpretation.

Key Insights and Ideas

  • Performance Doesn't Generalize: Strong performance on open-domain benchmarks does not guarantee alignment with expert preferences in niche domains. Preference alignment is highly domain-specific.
  • Prompting Hits a Ceiling: While adding role and task context (P2 and P3) improves performance, prompting alone is insufficient for complex, scenario-specific queries.
  • Structure Breeds Precision: Forcing LLMs to interact with structured serialized inputs (like JSON) yields better semantic precision and interpretability, likely due to cleaner token-level parsing.
  • Factual vs. Inferential Split: Factual questions (T1) rely heavily on direct document lookup, meaning task-level context provides marginal benefits. Conversely, inferential questions (T2) demand deep procedural reasoning and situational awareness, benefiting immensely from structured scaffolding.

Key Results

The authors evaluated a wide range of models, from compact open-source options to larger architectures.

For T1 (Factual) Questions, Qwen3 (32B) consistently outperformed all other models, achieving the highest semantic alignment (Cosine Similarity of 0.4302–0.4344). Smaller models like Llama 3.2 (1B) performed the worst. Interestingly, moving from role-aware (P2) to task-aware (P3) prompting provided almost no benefit for these straightforward factual queries.

For T2 (Inferential) Questions, the story changed. All models saw clear performance boosts when moving to task-aware prompting (P3). Mistral 7B Instruct emerged as the top performer for applied inference, achieving the highest faithfulness (0.8014) and semantic similarity (0.8336). Compact models like Phi-4-mini-instruct also showed substantial gains in context precision and faithfulness when given structured, task-aware prompts.

Limitations and Future Work

The primary limitation acknowledged by the authors is CANDI-QA's current reliance on static prompting. The framework lacks adaptive reasoning loops or real-time feedback mechanisms.

To address this, future work will focus on integrating self-reflective or planning-based modules capable of handling multi-hop reasoning dynamically. The authors also plan to expand the CANDI-QA dataset to cover a broader spectrum of MTSS scenarios beyond behavioral health.

Summary updated

Abstract

The deployment of large language models (LLMs) in specialized domains like medical diagnostics and financial advisory necessitates evaluating capabilities beyond general knowledge. Traditional question-answering benchmarks often fail to capture the nuanced contextual grounding, user awareness, and domain understanding these fields require. To address this, we introduce CANDI-QA (Contextual Alignment for Niche Domains Question Answering), a novel dataset evaluating LLMs on delivering accurate, context-sensitive, and user-aligned answers in specialized settings. CANDI-QA features expert-curated question-answer pairs structured into two categories: (1) Information Assistance Questions, which are direct, factual queries requiring precise extraction, and (2) Applied Inference Questions, which are multi-hop reasoning tasks needing situational inference to generate actionable insights. We evaluate over ten diverse language models, from compact open-source to state-of-the-art proprietary systems. As a robust baseline, we present MTSS-Net, a lightweight neuro-symbolic framework combining neural retrieval with rule-based reasoning. Our findings highlight the profound challenges of achieving contextual alignment in niche domains, revealing the limitations of current LLMs without enhanced contextual or symbolic integration. Ultimately, CANDI-QA serves as a critical benchmark for advancing research in context-aware language models, stimulating the development of robust, trustworthy AI for high-stakes domains.
BibTeX
@misc{chakraborty2026candi,
  author = {Megha Chakraborty and Darssan L. Eswaramoorthi and Het Riteshkumar Shah and Madhur Thareja and Michelle A Ihetu and Harshul Raj Surana and Kaushik Roy and Amit Sheth},
  title = {{CANDI: Contextual Alignment for Niche Domains Question Answering}},
  year = {2026},
  eprint = {2607.11891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.11891},
}
Figure