NEIGHBOR: Aligning Service Behaviour and Evidence Use in an LLM Agent for Community Resident Services

Zongyou Yang, Yinghan Hou, Xiaokun YangarXiv 2026 · arXiv

TL;DR

NEIGHBOR is a specialized LLM agent designed for community resident services that moves beyond simple system prompts by baking "service behavior" directly into the model via fine-tuning. By aligning model conduct with six specific service targets—like commitment control and human referral—the researchers achieved a massive 1.88-point jump in expert-rated quality over an un-tuned agent using the exact same tools.

What Problem Does This Paper Address?

When a resident contacts a community service desk about a mysterious service charge or a broken pipe, they don't just need a "smart" answer; they need an answer they can act on. This requires two things: grounded facts (from personal records or local policy docs) and appropriate conduct (knowing when to apologize, when to refer to a human, and never making promises the system can't keep).

Current LLM agents typically handle "conduct" through long system prompts. However, prompts can be brittle, and standalone models often "hallucinate" authority—promising refunds or repairs they aren't authorized to trigger. The authors tackle the challenge of creating a reliable, "agency-preserving" agent that stays within its lane while providing high-utility, empathetic support.

Methods

The researchers developed NEIGHBOR, which adapts a massive Qwen3-235B-A22B model using LoRA (Low-Rank Adaptation).

The core innovation is training the model on six service-behaviour targets:

  1. Acknowledgement: Recognizing the resident's concern without inventing details.
  2. Procedural Clarity: Providing ordered, actionable next steps.
  3. Evidence-linked Wording: Attributing facts specifically to records or guides.
  4. Commitment Control: Staying within the assistant's authority (no fake promises).
  5. Scope Awareness: Identifying when information is missing.
  6. Human Referral: Directing residents to staff when the AI hits its limit.
Figure 1: NEIGHBOR overview, traced with the synthetic billing example of Section 3. (a) Offline, public service materials become instruction data written to six behaviour targets, and LoRA SFT adapts Qwen3-235B-A22B into the NEIGHBOR model. (b) Online, the same model plans an evidence route, synthesises an answer that cites the period-T record, and proposes follow-up questions, from which the resident picks the next one to ask. Blue marks the fine-tuned model, green the case evidence, yellow the resident, and lavender the web-search fallback.

At runtime, the same fine-tuned model plays three roles: Planner (deciding whether to use SQL for records or RAG for documents), Synthesizer (writing the grounded answer), and Follow-up Predictor.

Datasets

The team curated a dataset of 13,215 instruction records derived from public community-service materials (FAQs, notices, reference tables) from a district in an East Asian city of 10 million people. From this, 7,824 SFT (Supervised Fine-Tuning) examples were used for training. Evaluation was performed on 300 held-out test queries, with a stratified subset of 50 queries subjected to rigorous human expert review.

Figure 2: Data and evaluation design. Public service materials become 13,215 instruction records and a 7,824-example SFT dataset. The 300 held-out test queries feed the seven-configuration comparison, and the expert rates all seven answers to a stratified 50-query subset.

Key Insights and Ideas

  • Behavior is a Fine-Tuning Target: We often think of fine-tuning for knowledge, but fine-tuning for conduct (how the model delivers that knowledge) is incredibly effective. It provides a much larger performance boost than simply adding tools to a base model.
  • The "Red-Line" Safety Net: Standalone LLMs (even huge ones) failed "red-line" tests 8-14% of the time by hallucinating policies or making unauthorized commitments. Integrating tools and fine-tuning for behavior reduced this rate to zero.
  • The LLM-as-Judge Trap: While an LLM judge (MiniMax-M2.7) ranked the models similarly to the human expert, it consistently underscored tool-grounded answers. Why? Because the agent often provided more specific, correct details (like office hours) than were in the "gold" reference answer. The LLM judge flagged these as "fabrications," whereas the human expert recognized them as helpful, grounded truths.

Key Results

NEIGHBOR-Full significantly outperformed all baselines. In an expert review on a 10-point scale:

  • NEIGHBOR-Full scored 7.74, compared to 5.86 for the un-tuned agent baseline.
  • The fine-tuning step alone added 1.88 points of quality, more than double the gain provided by adding the tools themselves (+0.76).
  • The model saw massive gains in Utility (+2.76), Retrieval (+2.60), and Empathy (+2.52).
Figure 3: Expert weighted scores after the red-line cap. Hatched segments show the red-line penalty and RL the red- line rate; both fall only on standalone models. NEIGHBOR- Full leads the agent baseline that shares its tools by 1.88 points (paired Wilcoxon, 𝑛= 50).

As shown in the dimension breakdown, the fine-tuned model didn't just get "smarter"—it became more helpful and better at citing its sources.

Figure 4: Where the gain occurs. (a) Expert dimension means; NEIGHBOR-Full has the highest mean on every dimension. (b) Gain of NEIGHBOR-Full over the agent baseline, in rating points (each 𝑝< 0.001, paired Wilcoxon, 𝑛= 50).

Limitations and Future Work

The authors note that evaluating tool-using agents requires "evidence-aware" rubrics. Standard reference-based evaluation fails when an agent is too good at finding supplementary facts.

Future work could involve expanding the "read-only" nature of the agent. Currently, NEIGHBOR cannot actually book a repair or issue a refund—it can only tell you how to ask a human to do it. Moving toward "write" capabilities (agency) will require even stricter commitment controls and multi-step verification to maintain resident trust.

Summary updated

Abstract

Residents who contact a community service desk about a repair or a service charge need answers they can act on, with facts that hold for their case and commitments the desk can keep. In tool-augmented LLM agents, such facts come from documents and records, and the conduct of a service answer is usually specified in a system prompt. We present NEIGHBOR, an LLM agent for community resident services that trains this conduct into the model that drives its tools. NEIGHBOR is deployed on a live resident-service platform for a district of an East Asian city of more than ten million people. It adapts Qwen3-235B-A22B with LoRA on 7,824 supervised fine-tuning examples built from public community-service materials around six service-behaviour targets, among them commitment control and human referral, and the adapted model plans document retrieval and read-only text-to-SQL over the resident's own records. In an offline evaluation with seven configurations, an independent domain expert rated all 350 answers to 50 stratified held-out queries. NEIGHBOR scores 7.74/10 against 5.86 for the same agent running the un-tuned model (+1.88; paired Wilcoxon $p<10^{-8}$; higher on 46 of 50 queries) and improves all six rated dimensions. No tool-augmented answer received a red-line flag (standalone models: 8-14% of answers). An LLM judge ranks the seven configurations much as the expert does (Spearman $ρ=0.93$). Its rubric counts specifics absent from the reference answer as fabricated, so it scores tool-grounded answers 1-2 points lower, enough to change which configuration ranks first. For resident-facing agents, service behaviour is a practical fine-tuning target that raises expert-rated answer quality with the tools held fixed.
BibTeX
@misc{yang2026neighbor,
  author = {Zongyou Yang and Yinghan Hou and Xiaokun Yang},
  title = {{NEIGHBOR: Aligning Service Behaviour and Evidence Use in an LLM Agent for Community Resident Services}},
  year = {2026},
  eprint = {2611.01836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01836},
}
Figure