COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation

Tianyi Zhou, Dongrui Liu, Leitao Yuan, Jing Shao, Xia HuarXiv 2026 · arXiv

TL;DR

COLLEAGUE.SKILL is an automated system that distills messy, heterogeneous human traces (such as chat logs, emails, and documents) into structured, portable AI agent skills. By separating professional capabilities from interaction styles into a version-controlled, correctable package, it allows LLM agents to embody specific human expertise without resorting to opaque prompts or full identity simulation.

What Problem Does This Paper Address?

AI agents are increasingly expected to carry specific human context—whether that means mimicking a departing teammate's code review style, adopting a specialist's decision heuristics, or embodying a public thinker's mental models.

However, building these "person-grounded" agents is notoriously difficult. Human expertise and style are rarely written down as clean instruction manuals; instead, they are scattered across messy, unstructured data like Slack channels, email threads, and design documents. Current approaches typically rely on massive, opaque system prompts or hidden memory stores. These methods are difficult to audit, impossible to port easily across different agent platforms, and prone to conflating a person's actual procedural judgment with their surface-level communication tone.

Methods

The authors introduce COLLEAGUE.SKILL, an end-to-end pipeline that treats trace-to-skill distillation as a software artifact construction problem rather than an open-ended simulation task.

The core workflow consists of several key stages:

  • Trace Intake & Routing: The system ingests unstructured data (Slack logs, PDFs, screenshots, WeChat exports) and normalizes them into local knowledge directories. It utilizes domain-specific presets—Colleague (enterprise/work traces), Celebrity (public writings/interviews), and Relationship (private interpersonal traces)—to specialize the underlying prompts and governance rules.
  • Dual-Track Distillation: Instead of dumping everything into a single prompt, the pipeline splits the distillation into two distinct tracks: a capability track (capturing workflows, technical standards, and decision heuristics) and a behavior track (capturing communication style, expression preferences, and interaction boundaries).
  • Artifact Generation: The system packages these tracks into a standardized, version-controlled format centered around a SKILL.md file, accompanied by machine-readable metadata (manifest.json and meta.json) for easy installation across agent hosts like Claude Code or OpenClaw.
  • Natural Language Evolution: Recognizing that initial generations are imperfect, the system features a correction handler. Users can provide natural language feedback (e.g., "she would push back here"), which the system automatically converts into Markdown patches or structured correction records {scene, wrong, correct}, incrementing the package version while maintaining a full rollback history.
Figure 1: COLLEAGUE.SKILL architecture for automated person-grounded skill generation. The shared distillation core renders portable agent-skill artifacts; domain presets add source requirements, evidence checks, consent assumptions, and lifecycle or gallery metadata.

Datasets

The paper does not utilize standard machine learning datasets for model training. Instead, the system operates directly on unstructured, user-provided human traces from real-world communication and documentation platforms, including Feishu, DingTalk, Slack, WeChat SQLite exports, email archives, PDFs, and direct text pastes.

Key Insights and Ideas

  • Artifacts Over Cloning: The authors explicitly reject the goal of full "behavioral cloning" or identity replacement. Instead, they frame the task as creating a bounded, editable technical artifact. The agent isn't trying to be the person; it is executing a package of distilled practices.
  • The Capability/Persona Split: Conflating factual knowledge and procedural judgment with surface tone is a major failure mode in persona systems. By separating them into work.md and persona.md, COLLEAGUE.SKILL allows users to inspect them independently or even invoke a "work-only" mode when stylistic mimicry is inappropriate.
  • Code-Like Lifecycle: By treating AI traits like software packages—complete with versions, patches, manifests, and rollbacks—the system makes person-grounded knowledge accountable and auditable.

Key Results

The system has been fully implemented as an open-source project and has achieved significant community adoption:

  • Repository Traction: At the time of writing, the public repository has garnered approximately 18.5k GitHub stars.
  • Ecosystem Scale: The public gallery features 215 skills contributed by 165 community members, with over 100k cumulative stars across the listed skill cards.
  • Interoperability: The generated skills conform to the emerging Agent Skills standard, proving successful deployment and invocation across multiple independent agent host environments.

Limitations and Future Work

  • Fidelity & Performance Evaluation: The paper focuses entirely on the engineering framework and artifact design. It does not provide quantitative evaluations of downstream task performance or behavioral fidelity, leaving these open for future human-subject studies.
  • Editor Bias: The natural language correction mechanism can inadvertently introduce the subjective bias of the person editing the skill, potentially smoothing over contested or nuanced human behaviors.
  • Privacy and Consent: While the local-first architecture mitigates data exposure, sensitive applications (like the relationship preset) introduce severe risks regarding non-consensual simulation and private data retention, requiring strict, separate governance frameworks.

Summary updated

Abstract

LLM agents are increasingly expected not only to complete isolated tasks, but also to carry bounded representations of human expertise, judgment, and interaction style. Building such person-grounded agents remains difficult because actionable knowledge associated with a person or role is usually embedded in heterogeneous traces rather than written as clean instructions. Existing memory and persona systems capture fragments of this evidence, while skill frameworks provide portable packaging formats; however, there is no end-to-end workflow for distilling these traces into inspectable, correctable, and agent-usable skills. We present an automated trace-to-skill distillation system for generating person-grounded AI skills via expert knowledge distillation. Given materials from a target person or role, COLLEAGUE.SKILL produces a versioned skill package with two coordinated tracks: a capability track for practices, mental models, and decision heuristics, and a bounded behavior track for communication style, interaction rules, and correction history. The package can be inspected, invoked, updated through natural-language feedback, rolled back, installed across agent hosts, and optionally prepared for controlled distribution. We describe the artifact contract, generation workflow, correction lifecycle, deployment surface, and domain presets implemented in the open-source system. At the time of writing, the public repository has approximately 18.5k GitHub stars; the gallery lists 215 skills from 165 contributors and more than 100k cumulative stars across listed skill cards. The system illustrates how person-grounded skills can be represented as portable, correctable packages rather than opaque prompts or hidden memories.
BibTeX
@misc{zhou2026colleague,
  author = {Tianyi Zhou and Dongrui Liu and Leitao Yuan and Jing Shao and Xia Hu},
  title = {{COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation}},
  year = {2026},
  eprint = {2605.31264},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.31264},
}
Figure