Choosing Principal Components by Cross-View Agreement for Multi-View Clustering

When the same objects are described in several ways, for example news stories covered by three outlets or images described by color, texture and shape features, clustering them usually starts by compressing each description with principal component analysis (PCA). PCA is a projection that keeps the directions along which the data varies most. The trouble is that "varies most" and "separates the groups" are different things. Large-variance directions often record lighting, background, source style or other nuisances, and the directions that actually separate clusters can sit far down the variance ranking. We propose to pick each view's principal components by a different test. A component is kept if the cluster structure found by the other views explains its variation. Because the other views never saw the component, this gives a label-free usefulness signal. The procedure alternates two steps: score components against a consensus built from the other views, then rebuild the consensus from the components that scored well.

This turns a known supervised result into an unsupervised, multi-view one. Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection showed that a handful of non-consecutive eigencomponents beats top-variance PCA on deep features. It found them by greedy search scored with AUROC on labeled anomalies, and a clustering problem has no such labels. Multiple views supply the missing supervision, provided we avoid the obvious trap of scoring components against a consensus they helped create.

Core Claim

For view $v$, let $z^{(v)}_j \in \mathbb{R}^n$ be the centered scores of the $n$ samples on its $j$-th principal component. Let $F^{(-v)} \in \mathbb{R}^{n \times c}$ be an orthonormal spectral embedding of $c$ clusters, computed from every view except $v$. The component's cross-view relevance is

$$s^{(v)}_j = \frac{\lVert F^{(-v)\top} z^{(v)}_j \rVert^2}{\lVert z^{(v)}_j \rVert^2},$$

which is the fraction of that component's variance lying in the span of the other views' cluster structure. The ratio does not depend on the component's scale, so it does not reward high variance. A large nuisance direction scores near zero unless the other views happen to share the nuisance. A small direction that tracks cluster membership scores near one.

We claim three things. First, selecting components by $s$ gives better clusterings than top-$k$ PCA per view. Second, the gain is largest exactly when high-variance nuisance directions are present. Third, the leave-one-view-out construction of $F^{(-v)}$ matters. Scoring against a consensus that includes view $v$ will tend to reinforce whatever that view contributed at initialization.

Research Program

The method alternates three blocks, in the spirit of the block-coordinate schemes surveyed in the file.

In the first block, each view is PCA-decomposed once, with mean removal before projection following Effective Dimensionality Reduction for Word Embeddings. We keep a generous pool of components, far more than the final count.

In the second block, given soft component weights $w^{(v)}_j = \mathrm{softmax}_j(s^{(v)}_j / T)$, each view builds an anchor graph on its reweighted components. An anchor graph links each sample to $m \ll n$ representative points, so memory and time stay at $O(nm)$. The views are then fused with the parameter-free weighting of AMGL, which weights each view's graph by the inverse of how poorly it fits the shared embedding $F$. This yields both the full consensus and the $V$ leave-one-out consensuses $F^{(-v)}$. Each leave-one-out consensus costs one small singular value decomposition of an $n \times m$ matrix.

In the third block, the scores $s$ are recomputed against the new $F^{(-v)}$, and the process repeats.

The exploration question enters through the temperature $T$. Hard selection at low $T$ commits early to one set of components. High $T$ keeps all components in play but blurs the selection. Adaptive exploration-exploitation schedules from swarm optimization do exist. Examples are the cosine-oscillated control factors in the Improved Barnacle Mating Optimizer and fitness-feedback inertia weights such as Hou et al.. They show that non-monotone schedules can escape poor basins, but they add tuning parameters with no guarantees, and they were developed for low-dimensional parameter search.

We therefore test one feedback rule, keeping it only if it earns its place. $T$ decays geometrically, and it is reheated when two conditions hold together: the partition has stopped changing, and the average cross-view agreement between leave-one-out consensuses is still low. The first condition says the algorithm has settled. The second says the views disagree about where it settled. This is the one situation where restarting the selection is informative rather than wasteful. We treat this schedule as a secondary contribution. If equal-compute random restarts match it, we drop it.

The role of prior work is as follows. Gula and Bertoldo and Simple and Effective Unsupervised Redundancy Elimination motivate the idea from opposite sides. The first shows that variance and usefulness diverge. The second shows unsupervised PCA working well when they align, with information plateauing near 200 of 768 dimensions. Our method should recover plain PCA's behavior in the second case and beat it in the first.

AMGL and MVCAN weight whole views. MVCAN additionally decouples per-view parameters so that a noisy view cannot dominate the shared gradients. We move the weighting down to individual components, so a noisy view can still contribute its few good directions.

DRMC refines anchor graphs with discrete label feedback, but its sample-level graphs cost $O(n^2)$. It also never touches the input dimensions, which is where our method acts, and at $O(nm)$ cost. GMLKM filters the spectra of precomputed base partitions to concentrate signal in the leading eigenvalues. It operates after clustering, whereas we operate before graph construction. The two are compatible, and we will test stacking them.

One honest threat comes from views that share a nuisance, such as all image descriptors being sensitive to the same illumination. In that case cross-view agreement rewards the nuisance. We measure this directly rather than assume it away.

Baselines

The naive reference point is concatenating all views, keeping the top-$k$ principal components, and running k-means. The file's direction summaries argue this fails because of mismatched scales and noise levels, but we will report it with $k$ tuned on the same grid as our method. If it comes close, our problem is milder than claimed.

The strongest structural competitor is per-view top-$k$ PCA feeding the same anchor-graph AMGL fusion we use. Here $k$ is tuned per dataset by the best unsupervised objective value, never by labels. This isolates exactly one change, which components are kept. We also report plain AMGL on raw features. On MSRC-v1, AMGL reports NMI 0.7432, against 0.7197 for MVSC. To count as beating AMGL, we must exceed its NMI on MSRC-v1 across ten seeds.

The strongest published systems in the file are DRMC, which reports 83.43% accuracy on 3sources, and GMLKM, which reports 87.2% accuracy on the Lung dataset against 86.7% for the next best method. We will rerun both from released code where available, with each method's own tuning protocol and the same data splits. We will report whether we match them at lower cost, or exceed them outright.

MVCAN is the reference for noisy views. It reports gains of up to 7% accuracy and 6% NMI on noise-simulated data, but the file gives no absolute accuracy for it on DHA, Caltech or NoisyBDGP. On those datasets, beating it means higher accuracy under the identical noise protocol with a linear-algebraic method using far less memory.

Two ablations isolate the key ingredients. The first scores components against the full consensus, including the view itself. The second uses a label-oracle greedy selection in the style of Gula and Bertoldo. This is an upper bound, not a competitor, and it shows how much of the achievable gain the label-free score recovers.

Why not just…?

Why not just use per-view top-$k$ PCA and tune $k$? This is the real threat on clean benchmarks. Ma et al. show top-$k$ PCA keeps about 99% of mutual information at 256 dimensions for text embeddings. We expect little gain on views like that. Our claim rests on the nuisance-injection experiment below and on image-feature views. There, Gula and Bertoldo found that adding consecutive components first plateaus and then hurts. If tuned top-$k$ matches us in the nuisance setting too, the core claim fails.

Why not just weight whole views, as AMGL and MVCAN do? A single scalar per view cannot keep the good part of a view and discard the rest. Our construction can. The decisive test injects large-variance noise into only some coordinates of one otherwise informative view. View-level weighting must either down-weight the whole view or absorb the noise. Component-level selection should do neither.

Why not just run the plain alternating method from many random starts and keep the best objective? This is the natural replacement for any exploration schedule, and it may well suffice. We run restarts under an equal budget of singular value decompositions and select among them by the unsupervised objective. The reheating schedule survives only if it wins on both final accuracy and seed-to-seed variance.

Why not just score components against the view's own clustering? This is the in-sample ablation. We expect it to lock in its own initialization. We will measure this as agreement between the selected component sets and the components that dominated the first iteration.

Proposed Experiments

We evaluate on MSRC-v1, Handwritten Numerals, 3sources, BBC4, Caltech101, AwA and the Lung dataset used by GMLKM. On each we report accuracy, NMI and purity over ten seeds, along with wall-clock time and peak memory. AwA, with more than 30,000 samples, tests the $O(nm)$ scaling argument against DRMC.

The decisive controlled experiment takes MSRC-v1 and Handwritten Numerals and adds to one view synthetic directions whose variance exceeds the view's leading principal component but which carry no class information. We sweep their number and magnitude. The method should hold its accuracy as the nuisance grows, while per-view top-$k$ PCA should degrade.

A second diagnostic records the variance ranks of the selected components. If nearly all selected components fall in the top ranks, the method has reduced to PCA, and the evidence for the "variance is not usefulness" story is weak for these data.

A third experiment repeats the injection with a nuisance shared across all views. This probes the known failure mode of cross-view agreement, and we will report it whatever the outcome.

Finally, we compare the leave-one-out scores with the label oracle's selections by measuring the overlap of the chosen component sets. This tells us how much of the supervised gain the unsupervised signal recovers.