Vocational colleges have to decide every year which skills to add to a training program and which to drop. Increasingly they make that call with a Delphi study, a structured expert survey in which panelists rate items anonymously over several rounds and see the group's summary scores between rounds until their ratings converge. The trouble is that nobody checks whether the consensus was right. A Delphi study of electric-vehicle technician competencies, for example, reports that its 15 experts converged on 106 indicators. It also says explicitly that it never tested the framework against what happened in the field (A Competency Framework for Electric Vehicle Maintenance Technicians). The training reforms that treat industry certifications as a "living syllabus" have the same gap: they report adoption and satisfaction, not whether the skills they chose were the ones employers went on to demand.
This proposal turns curriculum Delphi into a forecasting problem that can be scored. Panels of enterprise mentors and vocational teachers would rate, at a fixed date, how much each candidate competency will be demanded 12 and 24 months later. Those ratings would then be compared with skill demand automatically extracted from job postings published after that date. With a ground truth in hand, we can ask three questions neither literature can currently answer. Do expert panels add anything beyond simply extrapolating posting trends? Do Delphi's extra feedback rounds make forecasts more accurate, or only more uniform? And can panels of language-model agents stand in for human panels?
Core Claim
Our hypothesis is specific. On competencies that are already established, trend extrapolation from job postings will match or beat any expert panel. Expert Delphi will add measurable forecasting value only in two places: emerging technical competencies that have barely appeared in postings yet, and the professional-attitude and personal-trait competencies where human panels disagree most. We also predict that the drop in rating spread between rounds, which Delphi studies routinely report as evidence of consensus, partly reflects herding. If so, items that converge strongly will not be forecast more accurately than items that converge weakly, unless the feedback shown between rounds includes evidence from postings rather than only peer medians.
This differs from the closest work in three ways. The EV-technician study builds a static framework and never scores it. An LLM-based Delphi Study to Predict GenAI Evolution runs five-round panels of GPT-4o-mini agents and reports 100% thematic consistency across 12 configurations. That is a measure of agreement, not accuracy, and the study is about speculative technology forecasting with no ground truth. Skill-extraction work such as SkillSpan measures what postings say now but does not forecast and does not connect to curriculum decisions. Our contribution is the scoring harness and the head-to-head test: expert deliberation versus market text versus synthetic agents, all judged on the same future outcome.
Research Program
We would work in two sectors where the file documents both fast technical change and existing curriculum infrastructure. The first is electric-vehicle maintenance, which can start from the 106 indicators of the EV-technician framework and the 1+X-linked modules in Research on the Teaching Reform Strategies of New Energy Vehicle Majors. The second is intelligent-computing platform operations, starting from the tiered 1+X decomposition in Research on the reform of integrated talent cultivation of middle and high school undergraduate in the context of 1+X certificate and the Huawei-aligned courses in Research on the Collaborative Education Model between School and Enterprise for Computer Majors Based on Huawei Certification Standards.
The candidate pool for each sector combines three sources: the existing framework indicators, items drawn from the current certification syllabi, and items that a skill extractor surfaces from recent postings. The pool is frozen before any panel sees it, so no arm gets an unfair advantage.
The outcome measure is built from job postings collected continuously from the forecast date onward. A span extractor trained on SkillSpan pulls skill and knowledge phrases from each posting. These phrases are then matched to the frozen items by embedding similarity, with a human-audited sample used to estimate matching error. Extraction is noisy: the best SkillSpan models reach 56.64 F1 on skills and 63.88 F1 on knowledge (SkillSpan). To limit the damage, the outcome for each item is its prevalence across many postings rather than any single extraction. We also report how much the results change when estimated matching error is propagated through the analysis. The survey of computational job-market analysis warns that postings under-report implicit skills. For that reason, trait-type items are scored separately and backed by a small employer rating survey at 24 months.
Panels are recruited across at least three regions, deliberately including a less industrialized one. This addresses both the narrow US–Taiwan panel of the EV-technician study and the Shenzhen-only setting of Constraints, mechanisms, and strategies for industry-education integration in vocational education. That study identified enterprise willingness to cooperate as the binding bottleneck, so we cap enterprise effort at about one hour per round and log the hours actually spent.
Each panelist gives two things per item and horizon: a probability that the item's posting prevalence will rise by more than a pre-registered margin, and a rating of expected importance. This lets us score forecasts with standard probability-scoring rules as well as rankings.
Baselines
The naive reference point is persistence: assume every item's demand in 12 or 24 months equals its demand on the forecast date. This is what a curriculum that never changes implicitly bets on, and any method that cannot beat it has no case.
We expect the strongest competitor to be trend extrapolation from the same posting stream. This fits a per-item model of prevalence over the 24 months before the forecast date, ranging from a damped linear trend up to a small gradient-boosted model, and projects it forward. It gets the same posting data that the grounded panels are shown and the same tuning budget, with hyperparameters chosen by rolling-origin validation that uses only data from before the forecast date. No reported result for forecasting skill demand from postings was found on Lacuna, so this baseline's performance will be established here, not assumed. Beating it means one of two things: a statistically significant reduction in Brier score over all items, or a pre-registered gain in ranking accuracy (area under the curve) on the emerging-item subset where expert value is hypothesized. Ties on established items are expected and do not count against the claim.
The certification-follower baseline treats items added or reweighted in the latest vendor or 1+X syllabus revision as forecast to rise. It operationalizes the living-syllabus approach and is scored identically.
The LLM-agent Delphi baseline reimplements the pipeline of An LLM-based Delphi Study to Predict GenAI Evolution: an organizing agent that removes redundant questions by embedding similarity, persona-prompted responder agents, five rounds, and temperature 0.7. It receives the same item pool and the same posting summary as the grounded human arm. To avoid leakage, it must use models whose training cutoff precedes the forecast date. Its only reported figure is 100% thematic consistency across 12 runs, which says nothing about accuracy. We will therefore report its consistency and its forecast scores side by side.
The first ablation isolates feedback rounds. A one-round anonymous survey of the same panelists is compared with their own three-round Delphi, so any difference is attributable to iteration and peer feedback rather than panel composition. The second ablation isolates the feedback content. Two otherwise identical human Delphi arms differ only in what panelists see between rounds: peer medians and anonymous rationales alone, in the classic form described in An international, Delphi consensus study to identify priorities for methodological research in behavioral trials, or those plus a per-item posting-trend summary. A final hybrid arm adds evidence-citing AI co-participants, in the spirit of AI Agents and the Future of Deliberation. It is reported separately and does not count toward the core claim.
Why not just…?
Why not just extrapolate posting trends and skip the experts? This is the real threat, and the design is built to let it win. If trend extrapolation matches grounded Delphi on emerging and trait items as well as established ones, the honest conclusion is that curriculum committees should consume posting dashboards and stop convening panels. We will publish that result. The only reason to expect otherwise is that postings lag adoption and omit implicit skills, as the job-market survey notes. Whether that lag is large enough to matter at a 12-month horizon is an empirical question this study answers.
Why not just follow the vendor's certification updates? Vendors revise their exams for their own product lines. The Huawei-aligned curriculum paper itself flags lock-in, and the EV-technician study's attitude and trait dimensions do not appear in equipment exams. The certification-follower baseline tests this directly. We expect it to do well on vendor-specific technical items and poorly on cross-vendor and trait items, and the per-category scores will show whether that holds.
Why not just run the Delphi with LLM agents, which never tire? The GenAI-forecasting study reports a 15-month knowledge lag and a positivity bias. In our setting, a positivity bias should show up as over-predicting rises and therefore poor calibration, which the scoring will expose. If cutoff-safe agent panels given posting summaries match human panels, the human cost is unjustified for the dimensions where they match. We will report that dimension by dimension rather than overall.
Why not just hold a face-to-face advisory board? Boards produce minutes, not probabilities, so they cannot be scored this way. We will not run one as an arm. The one-round-versus-three-round ablation addresses the narrower question of whether structured iteration helps at all. If it does not, the argument for Delphi over cheaper consultation weakens accordingly.
Proposed Experiments and Decisive Checks
The main experiment runs every arm on the same frozen item pools in both sectors on a single forecast date, then scores them at 12 and 24 months. All arms are compared on Brier score, area under the curve for detecting rising items, and rank correlation between forecast and realized change in prevalence. Results are stratified by item type: established technical, emerging technical, and attitude or trait.
Three checks are decisive. The first is whether grounded human Delphi beats trend extrapolation on emerging items. The second is the herding test. For each item we regress forecast error on its round-to-round reduction in rating spread, the quantity the EV-technician study reports falling from 1.146 to 0.704 for personal traits. If convergence under peer-only feedback does not predict lower error, but convergence under posting-grounded feedback does, Delphi's consensus statistics should be redesigned. The third is cost per unit of accuracy. We record expert-hours per arm and plot Brier improvement over persistence against those hours. This addresses directly the enterprise-burden bottleneck that the Shenzhen study identified.
A secondary, curriculum-facing analysis translates each arm's forecasts into add-or-drop decisions under a fixed annual change budget. The budget reflects the tension between frequent refresh and teaching continuity described in Research on the Construction of the Professional Comprehensive Quality Training System for Excellent Field Engineers and Investigation of the Problems of Vocational Education in China. Decisions are scored by regret: items added that never gained demand, and items dropped that kept it.
References
- A Competency Framework for Electric Vehicle Maintenance Technicians: Addressing the Environmental, Social, and Governance (ESG) Imperatives of the BEV Industry
- An LLM-based Delphi Study to Predict GenAI Evolution
- SkillSpan: Hard and Soft Skill Extraction from English Job Postings
- Research on the Teaching Reform Strategies of New Energy Vehicle Majors from the Perspective of Industry-Education Integration
- Research on the reform of integrated talent cultivation of middle and high school undergraduate in the context of 1+X certificate
- Research on the Collaborative Education Model between School and Enterprise for Computer Majors Based on Huawei Certification Standards
- Deep Learning-based Computational Job Market Analysis: A Survey on Skill Extraction and Classification from Job Postings
- Constraints, mechanisms, and strategies for industry-education integration in vocational education: An empirical study
- An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research
- AI Agents and the Future of Deliberation: Designing Human-AI Collaboration for Democratic Dialogue
- Research on the Construction of the Professional Comprehensive Quality Training System for Excellent Field Engineers from the Perspective of New Quality Productive Forces
- Investigation of the Problems of Vocational Education in China