TL;DR
While Large Language Models (LLMs) excel at general knowledge, they often stumble in highly specialized, multi-stakeholder professional environments. This paper introduces CANDI-QA, a novel benchmark designed to evaluate "contextual alignment" within the niche domain of school behavioral health support, and proposes MTSS-Net, a neuro-symbolic baseline framework that demonstrates how structured, role-aware reasoning can significantly boost LLM performance where standard prompting fails.
What Problem Does This Paper Address?
When we talk about "AI alignment," we usually mean safety, factuality, or general human preferences. However, real-world enterprise and institutional settings require a different flavor of alignment: contextual alignment. This is a model's ability to tailor its responses based on the specific domain, the exact topic of conversation, and—crucially—the professional role and background knowledge of the user.
The authors focus on a highly complex, real-world use case: the Multi-Tiered System of Supports for Behavioral Health (MTSS-B) in schools.
As shown in Figure 2, MTSS-B involves a multi-disciplinary team (teachers, counselors, administrators, psychologists) operating across three distinct tiers of intervention (from school-wide prevention to intensive individual care). A generic LLM response to a behavioral health question might be factually correct but practically useless if it ignores whether the person asking is a school nurse or a principal, or whether the student in question requires Tier 1 or Tier 3 support. Traditional QA benchmarks completely miss this nuance.
Methods
To systematically evaluate and improve contextual alignment, the authors designed a dual-pronged approach:
1. Prompting Strategies
They tested ten different language models across three distinct prompting setups:
- P1 (No-Context): Only the user query is provided.
- P2 (Role-Aware): The system prompt includes the user's specific role and responsibilities.
- P3 (Task-Aware): The system prompt includes a high-level objective of the AI assistant alongside the role description.
2. Evaluation Framework
Because different questions require different types of thinking, the evaluation is split by question type:
- Factual Lookup (T1): Evaluated against expert "gold-standard" answers using traditional semantic and syntactic metrics (Cosine Similarity, BERTScore, MoverScore, BLEURT).
- Applied Inference (T2): Since these lack a single correct answer, the authors used a reference-free Retrieval-Augmented Generation (RAG) evaluation framework (RAGAs), measuring context precision, response relevancy, faithfulness, and semantic similarity against a strong reference context generated by a Qwen model.
3. The Baseline: MTSS-Net
To move beyond raw prompting, the authors introduced MTSS-Net, a lightweight neuro-symbolic framework:
- v1 (Structured Context): Encodes each QA instance into a structured JSON object capturing the task type, user role, intervention tier, and relevant policy references.
- v2 (Role-Aware Reasoning & Attribution): Adds curated expert QA pairs, explicit role conditioning, and source citations to ground the model's reasoning.
Datasets
The paper introduces CANDI-QA, a proprietary dataset meticulously curated with human experts from the School Behavioral Health Academy (SBHA).
As illustrated in Figure 3, the dataset consists of 315 total questions split into two categories:
- T1: Information Assistance Questions (90 questions): Direct, factual queries regarding best practices and evidence-based literature.
- T2: Applied Inference Questions (225 questions): Complex, multi-hop reasoning scenarios that are student- or situation-specific, requiring role-sensitive interpretation.
Key Insights and Ideas
- Performance Doesn't Generalize: Strong performance on open-domain benchmarks does not guarantee alignment with expert preferences in niche domains. Preference alignment is highly domain-specific.
- Prompting Hits a Ceiling: While adding role and task context (P2 and P3) improves performance, prompting alone is insufficient for complex, scenario-specific queries.
- Structure Breeds Precision: Forcing LLMs to interact with structured serialized inputs (like JSON) yields better semantic precision and interpretability, likely due to cleaner token-level parsing.
- Factual vs. Inferential Split: Factual questions (T1) rely heavily on direct document lookup, meaning task-level context provides marginal benefits. Conversely, inferential questions (T2) demand deep procedural reasoning and situational awareness, benefiting immensely from structured scaffolding.
Key Results
The authors evaluated a wide range of models, from compact open-source options to larger architectures.
For T1 (Factual) Questions, Qwen3 (32B) consistently outperformed all other models, achieving the highest semantic alignment (Cosine Similarity of 0.4302–0.4344). Smaller models like Llama 3.2 (1B) performed the worst. Interestingly, moving from role-aware (P2) to task-aware (P3) prompting provided almost no benefit for these straightforward factual queries.
For T2 (Inferential) Questions, the story changed. All models saw clear performance boosts when moving to task-aware prompting (P3). Mistral 7B Instruct emerged as the top performer for applied inference, achieving the highest faithfulness (0.8014) and semantic similarity (0.8336). Compact models like Phi-4-mini-instruct also showed substantial gains in context precision and faithfulness when given structured, task-aware prompts.
Limitations and Future Work
The primary limitation acknowledged by the authors is CANDI-QA's current reliance on static prompting. The framework lacks adaptive reasoning loops or real-time feedback mechanisms.
To address this, future work will focus on integrating self-reflective or planning-based modules capable of handling multi-hop reasoning dynamically. The authors also plan to expand the CANDI-QA dataset to cover a broader spectrum of MTSS scenarios beyond behavioral health.
Summary updated