Correlated Pixel Evidence for Nonparametric Object Counting in Video

Detection-free trackers built on Dirichlet process mixtures can discover and follow an unknown, changing number of objects in video without a trained detector. A Dirichlet process mixture is a clustering model that adds new clusters when the data demand it and charges a fixed price for each one. These trackers treat every foreground pixel as an independent draw from its object's model. This note argues that the independence assumption does more than blur shapes: it biases the object count itself. When $A$ correlated pixels are scored as $A$ independent pieces of evidence, the likelihood gain from splitting one object into two grows with image resolution, while the price of a new cluster stays fixed. The model therefore over-segments large, textured, or deforming objects, and its inferred count depends on how finely the camera samples the scene. We propose to replace the independent-pixel likelihood with a sparse Gaussian Markov random field over each object's pixel residuals. That field is a correlated Gaussian in which each pixel depends only on its neighbours. The correction keeps the evidence an object contributes on the same scale as the cluster-birth penalty.

The claim is concrete and falsifiable. With correlated per-object residuals, the inferred number of targets should be flat as resolution changes and should match ground truth more closely on scenes with large or articulated objects. The independent-pixel model should drift upward. The obvious cheap fix is to divide the independent log-likelihood by a single global "temperature". That fix is a serious competitor, and the experiments are built so that it can win. If it does, the honest conclusion is that tempering suffices.

Where This Sits

Supplying the tracking model, The Dependent Dirichlet Process Mixture of Objects for Detection-free Tracking and Object Modeling (DDPMO) and its precursor Unsupervised Detection and Tracking of Arbitrary Objects with Dependent Dirichlet Process Mixtures model each object as a spatial Gaussian plus a colour histogram whose parameters drift between frames, and they let clusters be born, persist through occlusion, and die. On an ant-tracking dataset DDPMO reports an Average Tracking Accuracy (ATA) of 0.47 against 0.19 for the next method. It also tracks about 100 T-cells with a near-correct count. Its pixel likelihood is a product of independent terms over foreground pixels. We keep its birth, death and transition structure and change only that likelihood. Its runtime of 4 to 20 seconds per second of video also motivates the scalable inference described below.

Supplying the correlation model, Learning Structured Gaussians to Approximate Deep Ensembles predicts a sparse Cholesky factor of a precision matrix over 5×5 neighbourhoods. It nearly matches an 8-model ensemble on KITTI depth (RMSE around 4.07 versus 3.85) and supports exact conditioning on new evidence. That work is static and has no notion of objects or counts. We borrow its sparse-precision parameterisation but attach one field to each mixture component rather than to the whole image.

Motivating the bias argument, A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation points out that summing independent pixel uncertainties overstates uncertainty about object geometry, because boundary errors move as contiguous blocks. We make the mirror-image point about evidence: summing independent pixel log-likelihoods overstates confidence in fine-grained partitions.

Supplying the inference route, DP-space uses small-variance asymptotics. It takes the limit of a Dirichlet process mixture as noise shrinks, which turns it into a fast, deterministic clustering objective with a per-cluster penalty. It reports about 3.3% error on Hopkins 155 in roughly 2 seconds, against more than 500 seconds for sparse subspace clustering. It works only offline, on linear subspaces, with Euclidean residuals. We derive the analogous objective with temporal birth and death terms and Mahalanobis residuals under a sparse precision.

Two supervised lines constrain the claims we can make. Bayesian Loss for Crowd Count Estimation With Point Supervision and Tracking Objects as Pixel-wise Distributions show that moving away from per-pixel independence improves counting and tracking when labels exist. P3AFormer reports 81.2% MOTA on MOT17. Both rely on annotations and fixed query slots, so they bound what is achievable with supervision and do not address unsupervised count discovery.

Core Claim and Mechanism

Consider a field with correlation length $\ell$ pixels covering $A$ pixels. It contains roughly $A/\ell^2$ effectively independent samples, which is the standard effective-sample-size argument for a stationary random field. The independent-pixel model credits it with $A$ samples instead. The log-likelihood improvement from fitting two clusters instead of one therefore scales with $A$, and the Dirichlet process birth penalty, of order $\log \alpha$, cannot compete once objects span many pixels. The prediction is that over-segmentation grows roughly with $\ell^2$, and hence with resolution and object size. Correlated residuals rescale the evidence to about $A/\ell^2$ and remove the dependence.

Taking the small-variance limit of a dependent Dirichlet process mixture with these residuals gives a per-frame objective:

$$\min_{z_t, K_t, \theta_t, Q} \; \lambda\, B_t + \delta\, D_t + \sum_{k=1}^{K_t} \Big( r_k^\top Q_k\, r_k - \log\det Q_k \Big) + \gamma \sum_{k} \lVert \theta_{k,t} - \theta_{k,t-1} \rVert^2 .$$

In this objective, $z_t$ assigns pixels to clusters and $K_t$ is the current number of clusters. $B_t$ and $D_t$ count births and deaths in frame $t$, and $\lambda$ and $\delta$ are their prices. $\theta_{k,t}$ holds the shape, colour and motion parameters of object $k$. $r_k$ is the vector of residuals between the features of the pixels assigned to $k$ (colour and optical flow) and that object's mean field. $Q_k$ is a sparse precision matrix over those pixels' 5×5 neighbourhood graph, controlled by a small number of per-object parameters such as a correlation range and a marginal scale. The log-determinant term keeps $Q_k$ from collapsing to zero. With $Q_k$ set to a scaled identity and the log-determinant dropped, the objective reduces to a deterministic, DP-means-style version of DDPMO.

Inference alternates three steps. Pixels are reassigned with a local move that respects the neighbourhood graph. $Q_k$ is refit by sparse Cholesky factorisation. Births and deaths are proposed where the residual cost exceeds $\lambda$. A full sequential Monte Carlo version with the same likelihood is run on short clips to check that the asymptotic objective preserves the posterior's count behaviour.

Research Program

The work has three parts. The first is analytic. We derive the scaling of the split-versus-merge likelihood ratio under independent and correlated residuals. We verify it on synthetic video, where blobs are rendered from known Gaussian random fields at controlled correlation lengths, resolutions and object counts. The second is the tracker itself: the streaming objective above, with optical flow as a pixel feature, so that the model needs only a coarse motion or foreground cue rather than DDPMO's background subtraction. The third is a set of ablations that place spatial dependence in different parts of the model. Dependence can be put in the likelihood (our proposal), in the prior over assignments (a Potts-style smoothing of labels), or in a scalar correction (tempering). The ablations ask which of these actually controls the count.

Baselines

The naive reference point is DDPMO with independent pixels, re-implemented inside our inference code with the same features, flow, and birth and death prices. Any gain then comes from the likelihood alone. On the ant data its published ATA is 0.47 from DDPMO, and our re-implementation must reproduce roughly that before any comparison is trusted. For PETS, the DDPMO and GPUDDPM papers report only rankings (tied second and third ATA), and no reported numerical ATA was found on Lacuna for PETS. We will report our re-run numbers for every method.

The strongest unsupervised competitor is the independent-pixel model with a tempered likelihood, where all pixel log-likelihoods are multiplied by one factor $\tau < 1$. It is tuned on a validation split with the same search budget we give to $\lambda$ and to our correlation-range prior. Beating it means lower count error and higher ATA on held-out sequences, and especially a flatter count-versus-resolution curve, since one fixed $\tau$ can be correct for only one resolution and object size.

The ablation that isolates our ingredient is a Potts-smoothed version. It adds a pairwise label-agreement prior over neighbouring pixels while keeping independent residuals. This tests whether spatial coherence in the assignments, rather than calibrated evidence, is what fixes the count. A further ablation, diagonal per-object variance, removes correlation while keeping per-object scale adaptation.

As a supervised reference rather than a target, we use Structured Learning for Cell Tracking, which reports 0.30% total tracking loss on DCellIQ and 93.2% mitosis detection on Mitocheck using pre-segmented cells and labelled training data. We do not expect to beat it. The relevant question is how much of the gap the correlated likelihood closes relative to independent DDPMO.

Why not just…?

Why not just temper the independent-pixel likelihood? This is the real threat. If correlation length were constant across objects and scales, one temperature would be exactly the right correction. Our argument is that $\ell$ varies with object texture, size and blur, so the needed temperature varies within a single frame. The synthetic experiments, where $\ell$ is mixed within scenes, and the multi-resolution tests on real data settle this directly. If a single tuned $\tau$ matches us within noise on held-out resolutions, we will report that tempering suffices and that the precision model adds cost without benefit.

Why not just add a spatial smoothing prior on the cluster labels? A Potts prior makes labels contiguous, but it does not change how much evidence each pixel contributes. A large textured object can still be split into two contiguous halves if that buys enough independent-pixel likelihood. The Potts ablation tests this. We predict it reduces speckle but leaves the resolution drift in the count. If it removes the drift, the mechanism we claim is wrong.

Why not just use a supervised or class-agnostic detector with a standard tracker? Where labels exist this is the right answer, as P3AFormer's 81.2% MOTA on MOT17 shows. The proposal targets the regime DDPMO targets: ants, T-cells and other domains with no trained detector. There, MOT16 baselines already reach only 33.7% MOTA even on pedestrians with public detections. We do not claim to compete on MOTChallenge.

Proposed Experiments

The first decisive check is the resolution sweep. Each test sequence (ants, PETS, T-cell microscopy, DCellIQ and Mitocheck frames) is rendered at four resolutions spanning a factor of four. We plot mean absolute per-frame count error and the slope of inferred count against resolution for every method. The hypothesis predicts a positive slope for the independent, Potts and diagonal models, a slope near zero for ours, and a tempered model that is flat only near the resolution it was tuned at.

The second check is the synthetic benchmark, where the true count and true $\ell$ are known. We measure count error as a function of $\ell$ and verify the predicted roughly quadratic growth of over-segmentation under independence.

The third check is end-task tracking: ATA and identity fragmentation on ants and PETS, and tracking loss and division detection on DCellIQ and Mitocheck. We use the same features, flow and tuning budget for every unsupervised method.

Two further measurements accompany these. Runtime per second of video is compared with DDPMO's reported 4 to 20 seconds. We also compare the asymptotic objective against the full sequential Monte Carlo version on short clips, to confirm that the speedup does not change count behaviour. As a sanity check that our small-variance machinery is sound, the static, isotropic special case run on Hopkins 155 should land near DP-space's reported 3.3% error.