Depth-Wise Task Clustering from Visual Feature Statistics

When a model learns a stream of tasks, such as segmenting polyps, then skin lesions, then breast ultrasound masses, it has to decide which earlier knowledge each new task should reuse. Bayesian non-parametric clustering is a natural tool for this decision. These methods use a prior, such as the Chinese Restaurant Process, that lets the number of task groups grow as new kinds of tasks appear instead of fixing it in advance. The strongest recent system in this line, MedCRP-CL, reached this answer in an unexpected way. It clusters tasks using CLIP embeddings of clinical text prompts, because clustering on image features "proved unreliable." Text turned out to be "a better compass than pixels." That finding matters because most image streams arrive with no descriptive text at all. If the model's own learned features cannot tell it which tasks are related, non-parametric routing is confined to settings that come with good metadata.

We propose that pixels failed as a compass because the image features were summarized at a single depth, not because images carry too little signal. A deep network's layers encode different things. Early layers respond to color, texture, and acquisition style, which is what separates endoscopy from ultrasound. Deep layers respond to shape and anatomy, which is what separates heart from breast. Two tasks can therefore be close at one depth and far apart at another. Our method runs a separate Chinese Restaurant Process at each depth block of a frozen backbone. Each process clusters tasks using second-order feature statistics computed at that block, and each cluster owns a low-rank adapter for that block only. Tasks end up sharing early adapters along one partition and deep adapters along another. The whole method uses no text, no task labels at test time, and no replay of old data.

Core Claim

Task-level summaries of feature statistics, taken separately at each depth, are reliable enough for non-parametric task discovery. Allowing a different partition at each depth gives better transfer per adapter parameter than a single partition. Concretely, we expect two results on the 16-task continual medical segmentation benchmark used by MedCRP-CL. First, the text-free method should come within about one Dice point of MedCRP-CL's text-routed result. Second, the partitions found at early blocks should align with imaging modality while those at deep blocks align with organ, recovering the benchmark's two ground-truth groupings (4 modalities, 5 organs) without being told either.

Relation to Prior Work

Several papers motivate the gap. MedCRP-CL routes tasks to modality-specific LoRA adapters (small low-rank weight updates added to a frozen backbone) using a Chinese Restaurant Process over text embeddings. We keep its adapter-plus-prior design but replace text with visual statistics and replace one partition with one per depth. Its negative result on pixel clustering is the claim we test head-on. Multi-Task Zipping merges independently trained CNNs neuron by neuron using Hessian similarity and finds that up to 90% of parameters can be shared across a five-task ResNet suite. This is evidence that the right amount of sharing depends on the layer. However, it merges after training and has no mechanism for tasks that arrive over time. Factorial Multi-Task Learning shows that partial sharing outperforms all-or-nothing grouping. It uses a Dirichlet process over task groups combined with a beta process over shared bases, and reports 6.99% versus 13.4% MNIST multi-task error against GO-MTL. It is restricted to linear models with Gibbs sampling, and its authors name deep integration as future work. Our per-depth partitions are a deep, online, much simpler form of that partial sharing.

Other papers constrain the design. A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning (CN-DPM) creates a whole classifier plus density-estimator expert for each cluster and routes inputs by generative likelihood. Its accuracy is limited by that density estimate, and its memory grows with each expert. Deep Dirichlet Process Mixture Models fixes the density problem with an invertible flow trained by Monte Carlo EM, but at Gibbs-sampling cost and only for single-dataset clustering. We take from both that raw high-dimensional densities are fragile. We avoid flows by clustering low-dimensional, shrinkage-regularized summaries instead of raw activations. LIMix grows components when a discrepancy bound is exceeded, but it relies on generative replay, which we exclude. Results from 3D brain MRI classification show deeper networks failing to beat shallow ones on 231 scans. That is a reminder that small medical tasks favor few trainable parameters, which supports adapters over whole experts.

One paper supplies a key technique. Efficient Parameter Isolation (EPI) identifies the task at test time using the Mahalanobis distance of pre-logit features to per-task Gaussians, and reports large gains over softmax-confidence and Euclidean routing. We reuse this Mahalanobis routing, but at every depth and over clusters rather than individual tasks. EPI gives every task its own prefix and never merges related ones.

Research Program

The backbone is a frozen segmentation encoder-decoder of the kind used in MedCRP-CL, divided into $L$ blocks. When task $t$ arrives with its training images, we pass them through the frozen backbone. At each block $l$ we compute a descriptor $d_t^l$: the mean and a low-rank projection of the channel covariance of activations, averaged over the task's images. Averaging over a whole task is deliberate. It removes the per-image noise that we suspect undermined pixel clustering. Each block runs its own Chinese Restaurant Process. Task $t$ joins existing cluster $k$ at block $l$, or opens a new one, with probability

$$ P(z_t^l = k) \propto n_k^l \, \mathcal{N}(d_t^l;\, \mu_k^l, \Sigma_k^l), \qquad P(z_t^l = \text{new}) \propto \alpha \, p_0(d_t^l), $$

where $z_t^l$ is the cluster assignment, $n_k^l$ is the number of tasks already in cluster $k$, $\mu_k^l$ and $\Sigma_k^l$ are that cluster's descriptor mean and shrinkage covariance, $\alpha$ is the concentration parameter that controls how readily new clusters open, and $p_0$ is the prior predictive under a Gaussian base measure.

Assignment is a single greedy step per task, so no MCMC is needed. If a task joins an existing cluster, it continues training that cluster's adapter at block $l$, with Elastic Weight Consolidation to protect earlier members. If it opens a new cluster, it starts a new adapter, initialized from the nearest cluster's adapter. At test time no task identity is given. Each block chooses its adapter by Mahalanobis distance between the image's own block statistics and per-cluster Gaussians, which are updated with running estimates. Training still sees task boundaries, as in MedCRP-CL. A boundary-free variant that detects shifts in the descriptor stream is a stated extension, not a claim of this proposal.

The analysis has three parts. We measure how well the discovered partitions match ground truth at each depth. We track how adapter count grows as $\alpha$ varies. We examine test-time routing errors separately for each block, to determine whether misrouting concentrates where density estimates are hardest. Prior work points to deep blocks as the likely trouble spot.

Baselines

The naive reference is sequential fine-tuning of the shared model with no isolation. On this benchmark MedCRP-CL reports a 28.3% forgetting rate for it, and any method in this space must sit far below that.

The strongest existing method is MedCRP-CL itself, which reports 73.3% average Dice and 4.1% forgetting on the 16-task benchmark (MedCRP-CL). It is the right competitor because it uses the same prior and adapter family but has access to text, which our method does not. We will rerun it in our setup with the same backbone, adapter rank, task orders, and three seeds. We will report both its published numbers and our reproduction. Our method gets the same hyperparameter search budget, with the search spent on $\alpha$ and covariance shrinkage. Since our method has strictly less information, coming within one Dice point at no worse forgetting counts as a success. Exceeding MedCRP-CL counts as evidence that per-depth sharing adds something text cannot.

Two further published baselines are MoE-Adapters, which MedCRP-CL outperforms by 8.0 Dice (an implied 65.3%) while using 6× fewer trainable parameters, and CL-LoRA, with 9.7% forgetting. Both are run under a matched trainable-parameter budget.

We also build an EPI-style baseline: one LoRA adapter per task, routed by Mahalanobis distance on final-layer features. This isolates the value of clustering itself. For it, no reported result was found on Lacuna on this benchmark.

The key ablation is a single-depth variant. It uses one Chinese Restaurant Process on final-block descriptors, with that one partition applied to all blocks. This is the closest visual analogue of MedCRP-CL. Further controls are a random partition with the same cluster counts, and an oracle that partitions by the true modality labels at every block.

Why not just…?

Why not just run one clustering on the frozen backbone's final embedding? This is the real threat, and the single-depth ablation tests it directly. If it matches the per-depth method, the paper's conclusion changes. The finding would then be that task-level averaging of second-order statistics, not depth, is what rescues pixel clustering. That still overturns the "text over pixels" claim, though in a narrower way. We expect per-depth partitions to win specifically on task pairs that share a modality but differ in organ, such as cardiac versus breast ultrasound, and we will report Dice on those pairs separately.

Why not just give every task its own adapter and route by Mahalanobis distance, as EPI does? Isolation removes interference, but it also removes transfer and makes parameters grow linearly with the number of tasks. Small medical tasks are where that hurts most, given that depth alone overfits at a scale of hundreds of samples (3D brain MRI). We will compare Dice on the smallest tasks in the stream, together with total adapter parameters. If per-task isolation wins on both, clustering is not worth its complexity.

Why not just use text or dataset metadata when it exists? When good prompts exist, MedCRP-CL shows that they work. Many streams lack them, however, including archival scans, industrial inspection, and unlabeled sensor feeds. Our comparison is designed to measure the cost of dropping text, not to argue that text should be dropped where it is available.

Why not just replay old data? Replay is weak on harder streams: reservoir sampling reaches 10.01% against CN-DPM's 20.10% on task-free Split-CIFAR100 (CN-DPM). Storing patient images is also often not permitted. We include replay only as a reference point outside the replay-free comparison.

Proposed Experiments

The main experiment uses the 16-task medical segmentation stream, with average Dice, forgetting rate, and trainable parameters as metrics, measured over three task orders. The decisive check has two parts. We compute the adjusted Rand index between each block's discovered partition and both ground-truth groupings, modality and organ. The hypothesis predicts that early blocks align more with modality and deep blocks more with organ. If the alignment is flat across depth, the core mechanism is wrong regardless of Dice.

A second check probes generality on the twelve 2D MedMNIST v2 datasets (MedMNIST v2) presented as a classification stream. No continual-learning result was found on Lacuna for this setup. The only reference is the per-dataset average AUC of 0.927 reported for Google AutoML Vision, which serves as a joint-training ceiling rather than a competitor.

A third check runs the single-depth and per-depth variants on Split-CIFAR100 with task boundaries given. This tests whether the effect is specific to medical modalities. It is not a head-to-head with CN-DPM's task-free 20.10%, because the settings differ.

Across all three, we sweep $\alpha$ over two orders of magnitude and report how cluster count and Dice vary. If performance swings sharply with $\alpha$, the method has moved the tuning burden rather than removed it.