Many prediction tasks output a bundle of related attributes at once: the 312 visual traits of a bird, the gender, age and clothing of a pedestrian, or the rubric scores of an essay. The standard design gives each attribute its own output head and predicts them all in parallel and independently. A newer line of work instead generates the attributes one at a time, so that each prediction can condition on the ones already made. This autoregressive approach (one prediction per step, each step reading the previous outputs) gives large gains in several settings. On the CUB birds benchmark, an autoregressive concept predictor reaches 81.4% concept accuracy versus 48.1% for an independent soft predictor (Addressing Leakage in Concept Bottleneck Models). It costs one sequential decoding step per attribute, though, and its results depend on the chosen order. For 312 concepts that means 312 sequential steps.
We propose to keep the parallel predictor and train it on the teacher's committed decisions: a single, internally coherent attribute configuration decoded by an autoregressive teacher for each input. The underlying reason is simple. A product of independent per-attribute heads represents a target exactly when that target is a deterministic function of the input. Mode-selected teacher outputs are close to deterministic, while noisy human annotations are not. Because these training targets are discrete rather than soft probabilities, the student's concepts also cannot hide task information in continuous values, which is the leakage failure that hard autoregressive concept models were built to avoid. We test whether this one recipe recovers the teacher's accuracy and joint coherence at parallel cost, and whether it keeps the intervention behavior that makes concept bottlenecks useful.
Core Claim
The claim is that sequence-level distillation with hard targets transfers most of an autoregressive attribute model's advantage into independent parallel heads. This works because the teacher removes the dependence among attributes that the independent factorization cannot represent, not because the student learns to imitate the conditioning. The quantity that matters is the conditional total correlation of the training targets,
$$\mathrm{TC}(y \mid x) = \sum_{i=1}^{N} H(y_i \mid x) - H(y_1,\dots,y_N \mid x),$$
where $y_i$ is the $i$-th of $N$ attributes, $x$ is the input, and $H$ is conditional entropy. Under log-likelihood training, $\mathrm{TC}$ is exactly the extra loss an independent-head model pays relative to the best joint model. Ground-truth labels with annotator noise and several valid configurations have positive $\mathrm{TC}$. Targets decoded from a teacher have $\mathrm{TC}$ near zero.
This mirrors the explanation Understanding Knowledge Distillation in Non-autoregressive Machine Translation gave for why distillation makes parallel translation work: the teacher picks one mode and lowers target complexity. That paper supplies our mechanism. Our contribution is to carry it to heterogeneous attribute sets, where it has not been tested. Attribute sets differ from translation in two ways that matter. Their structure comes from semantic and physical groupings rather than grammar. Some attribute sets also act as interpretable bottlenecks, where the form of the transferred signal matters.
Positioning
Four kinds of prior work bear on this proposal.
The first is the autoregressive attribute models, which motivate it by showing the benefit of conditioning. They also show its costs. Point2Seq decodes 3D boxes as location, then size, then category. SequencePAR decodes pedestrian attributes from CLIP features with a causal decoder. ArTS emits essay trait scores as T5 text. ARECHO chains speech metrics and randomizes the order during training so that any order can be used at test time. All four keep the sequential decoder at test time. ARECHO explicitly reports higher latency than parallel heads, and ArTS and Point2Seq depend on hand-chosen orders. We use these models only as teachers.
The second is the work that constrains our design. A Study on the Autoregressive and non-Autoregressive Multi-label Learning finds that autoregressive models break down on large unordered label sets: 27.88% versus 64.63% Precision@3 on Wiki10-31K. It also finds they keep an edge in top-1 precision on small sets. This warns us that a teacher with a fixed order may be a poor teacher when the label set is large. For that reason the teacher is trained with randomized orders, as in ARECHO.
The third is prior multi-label distillation. Multi-Label Knowledge Distillation (L2D) distills one-versus-all soft logits and per-label embeddings, raising a ResNet-34 student from 70.31 to 72.87 mAP on MS-COCO. It treats labels as conditionally independent and passes soft marginals, which is exactly what we argue cannot carry joint structure. We use it as a baseline.
The fourth is the leakage result. Addressing Leakage in Concept Bottleneck Models shows that soft concept probabilities smuggle task information and break human intervention. Its hard autoregressive model fixes this while staying within about 1% of the leaky soft models' label accuracy. This creates a direct conflict with classic soft-target distillation (Transferring knowledge from a RNN to a DNN). Our hard-target design is chosen to resolve that conflict rather than avoid it.
Research Program
Teacher. For each benchmark we train an autoregressive attribute teacher on a shared backbone. The teacher is trained with randomized attribute orders. Following ARECHO, we add a small number of fixed "anchor" orders where the domain suggests one, such as grouping mutually exclusive attribute families together.
Distillation targets. For each training input the teacher produces a hard joint configuration. We decode it with beam search under several orders and keep the configuration with the highest joint likelihood averaged across orders. Choosing the target this way means it no longer depends on any single order, which is a cheap answer to the ordering-sensitivity problem.
Student. The student is the plain parallel multi-head model on the same backbone. It is trained on a mixture of teacher targets and ground truth, with the mixing weight tuned on validation data.
Diagnostic. We estimate $\mathrm{TC}(y \mid x)$ for both the original and the distilled training sets, using the teacher's own joint likelihood as the joint model. We then test whether the student's remaining gap to the teacher tracks this quantity across datasets.
Capacity matching. We vary teacher capacity, following the finding that weak parallel students prefer weak teachers. In translation, switching the teacher from "base" to "small" raised a FlowSeq student from 18.55 to 22.0 BLEU (Understanding Knowledge Distillation in Non-autoregressive Machine Translation).
Testbeds. The primary testbed is CUB-2011 with its 312 binary concepts and downstream species label. Here the sequential cost is largest, ground-truth concepts are noisy, and leakage can be measured. The second is pedestrian attribute recognition on PETA, PA100K and RAPv2, where SequencePAR is the teacher. The third is ASAP/ASAP++ multi-trait essay scoring, where traits are ordinal and data-scarce traits such as Voice stress the teacher. 3D detection is deliberately left out. Point2Seq's gains there mix sequential conditioning with dynamic feature sampling, so the comparison would be confounded.
Baselines
The naive reference is independent parallel heads trained on ground truth with the same backbone, data and tuning budget as the student. It is the deployed default. If it already matches the teacher, the autoregressive gains reported in prior work came from other differences, such as backbone or training schedule, and our premise fails. We therefore re-implement it ourselves rather than quote the numbers from prior papers.
The strongest existing method is the autoregressive teacher itself. On CUB the reference is the hard autoregressive model's reported 81.4% concept accuracy (Addressing Leakage in Concept Bottleneck Models). On PETA it is SequencePAR's reported 84.92% accuracy and 90.46% F1 (SequencePAR). On ASAP/ASAP++, ArTS reports more than a 5% average QWK gain over baselines, but the file has no absolute per-trait QWK, so we will reproduce it. Beating the teacher is not the goal. Success is a student within one point of the teacher on concept accuracy, F1 or QWK, with decoding latency equal to a single parallel pass.
The strongest distillation competitor is L2D's one-versus-all soft-logit and label-wise embedding distillation (Multi-Label Knowledge Distillation). We apply it with the same autoregressive teacher, computing the teacher's marginals by sampling, and give it the same tuning budget. Our method must beat it on joint metrics and on leakage, not only on marginal accuracy.
The key ablation swaps our hard targets for soft marginals, keeping teacher, student and budget fixed. A second ablation uses single-order decoding instead of decoding averaged across orders. A third trains the teacher with a fixed order rather than randomized orders.
Why not just…?
Why not just train better independent heads? For metrics computed attribute by attribute, a well-calibrated independent head is in principle optimal, so this is a real threat. Our answer is empirical and comes in two parts. The first is the re-implemented naive baseline. The second is that we report joint metrics alongside marginal ones. These are exact-match rate over attribute groups, the violation rate of mutually exclusive families (CUB color groups, pedestrian age bins), and cross-trait consistency of essay score profiles. Independent training on noisy, high-$\mathrm{TC}$ labels should do poorly on exactly these measures. If the naive heads match the teacher on joint metrics too, we will report that conditioning was never the source of the gain.
Why not just distill soft marginals, as in L2D or classic distillation? Soft marginals carry no information about how attributes co-occur. Two teachers with identical marginals but different joint structure give identical targets. On concept bottlenecks, continuous concept values are also the documented leakage channel (Addressing Leakage in Concept Bottleneck Models). The hard-versus-soft ablation settles both points directly.
Why not just add one refinement pass, where parallel heads read each other's first-round predictions? This keeps the cost constant and partly captures dependence, so it is a serious competitor. We include a two-pass parallel model at matched parameters and FLOPs. We also test whether it composes with our method, since a refinement pass trained on distilled targets may be the best deployable system. In that case the result becomes "distillation plus one pass" rather than a claim that distillation alone is sufficient.
Why not just accept the autoregressive cost? For ten essay traits the cost is small, and we say so plainly. The case for parallel students rests on large attribute sets such as CUB's 312 concepts, on removing order dependence, and on the leakage-safe form of the targets. Latency is measured on every benchmark so readers can judge where the trade-off matters.
Proposed Experiments and Decisive Checks
On CUB we report concept accuracy, label accuracy, and wall-clock latency. We also report intervention curves, which show label accuracy as humans correct $k$ concepts. Leakage is measured as the gain of a probe that predicts the label from predicted concepts over the same probe applied to ground-truth concepts. The decisive outcome is a hard-distilled parallel student that reaches near-teacher concept accuracy, shows leakage comparable to the hard autoregressive model and clearly below soft distillation, and runs in a single pass.
We expect one honest cost. The teacher propagates a human correction to later concepts, and a parallel student cannot. We therefore report intervention curves for the student alone and for a hybrid that falls back to the teacher only for the concepts a human edits.
Across all three testbeds, the decisive mechanistic check is whether the per-dataset gap closed by distillation correlates with the drop in estimated $\mathrm{TC}$ between raw and distilled targets. A clear positive relation supports the claim that the teacher helps by removing dependence. No relation would mean the gains come from teacher denoising alone, which the independent-heads-on-cleaned-labels comparison can isolate.
Finally, the capacity-matching sweep tests whether attribute students, like translation students, prefer smaller teachers. A positive answer would give a practical rule for choosing teachers.
References
- Addressing Leakage in Concept Bottleneck Models
- Understanding Knowledge Distillation in Non-autoregressive Machine Translation
- Point2Seq: Detecting 3D Objects as Sequences
- SequencePAR: Understanding Pedestrian Attributes via A Sequence Generation Paradigm
- Autoregressive Score Generation for Multi-trait Essay Scoring
- ARECHO: Autoregressive Evaluation via Chain-Based Hypothesis Optimization for Speech Multi-Metric Estimation
- A Study on the Autoregressive and non-Autoregressive Multi-label Learning
- Multi-Label Knowledge Distillation
- Transferring knowledge from a RNN to a DNN