TL;DR
NEIGHBOR is a specialized LLM agent designed for community resident services that moves beyond simple system prompts by baking "service behavior" directly into the model via fine-tuning. By aligning model conduct with six specific service targets—like commitment control and human referral—the researchers achieved a massive 1.88-point jump in expert-rated quality over an un-tuned agent using the exact same tools.
What Problem Does This Paper Address?
When a resident contacts a community service desk about a mysterious service charge or a broken pipe, they don't just need a "smart" answer; they need an answer they can act on. This requires two things: grounded facts (from personal records or local policy docs) and appropriate conduct (knowing when to apologize, when to refer to a human, and never making promises the system can't keep).
Current LLM agents typically handle "conduct" through long system prompts. However, prompts can be brittle, and standalone models often "hallucinate" authority—promising refunds or repairs they aren't authorized to trigger. The authors tackle the challenge of creating a reliable, "agency-preserving" agent that stays within its lane while providing high-utility, empathetic support.
Methods
The researchers developed NEIGHBOR, which adapts a massive Qwen3-235B-A22B model using LoRA (Low-Rank Adaptation).
The core innovation is training the model on six service-behaviour targets:
- Acknowledgement: Recognizing the resident's concern without inventing details.
- Procedural Clarity: Providing ordered, actionable next steps.
- Evidence-linked Wording: Attributing facts specifically to records or guides.
- Commitment Control: Staying within the assistant's authority (no fake promises).
- Scope Awareness: Identifying when information is missing.
- Human Referral: Directing residents to staff when the AI hits its limit.
At runtime, the same fine-tuned model plays three roles: Planner (deciding whether to use SQL for records or RAG for documents), Synthesizer (writing the grounded answer), and Follow-up Predictor.
Datasets
The team curated a dataset of 13,215 instruction records derived from public community-service materials (FAQs, notices, reference tables) from a district in an East Asian city of 10 million people. From this, 7,824 SFT (Supervised Fine-Tuning) examples were used for training. Evaluation was performed on 300 held-out test queries, with a stratified subset of 50 queries subjected to rigorous human expert review.
Key Insights and Ideas
- Behavior is a Fine-Tuning Target: We often think of fine-tuning for knowledge, but fine-tuning for conduct (how the model delivers that knowledge) is incredibly effective. It provides a much larger performance boost than simply adding tools to a base model.
- The "Red-Line" Safety Net: Standalone LLMs (even huge ones) failed "red-line" tests 8-14% of the time by hallucinating policies or making unauthorized commitments. Integrating tools and fine-tuning for behavior reduced this rate to zero.
- The LLM-as-Judge Trap: While an LLM judge (MiniMax-M2.7) ranked the models similarly to the human expert, it consistently underscored tool-grounded answers. Why? Because the agent often provided more specific, correct details (like office hours) than were in the "gold" reference answer. The LLM judge flagged these as "fabrications," whereas the human expert recognized them as helpful, grounded truths.
Key Results
NEIGHBOR-Full significantly outperformed all baselines. In an expert review on a 10-point scale:
- NEIGHBOR-Full scored 7.74, compared to 5.86 for the un-tuned agent baseline.
- The fine-tuning step alone added 1.88 points of quality, more than double the gain provided by adding the tools themselves (+0.76).
- The model saw massive gains in Utility (+2.76), Retrieval (+2.60), and Empathy (+2.52).
As shown in the dimension breakdown, the fine-tuned model didn't just get "smarter"—it became more helpful and better at citing its sources.
Limitations and Future Work
The authors note that evaluating tool-using agents requires "evidence-aware" rubrics. Standard reference-based evaluation fails when an agent is too good at finding supplementary facts.
Future work could involve expanding the "read-only" nature of the agent. Currently, NEIGHBOR cannot actually book a repair or issue a refund—it can only tell you how to ask a human to do it. Moving toward "write" capabilities (agency) will require even stricter commitment controls and multi-step verification to maintain resident trust.
Summary updated