Codec-Neutral Compressed-Domain Cell Tracking for Bioreactor Monitoring

Video compressors already compute where each block of an image moved since the previous frame and how much the result differs from that prediction. A small line of vision research reads these by-products straight out of the compressed file instead of decoding every frame to pixels, and reports large speedups. The catch is that each such model is wired to one compression standard's internal layout, and newer standards chop images into blocks very differently. This note proposes a cell and microcarrier tracker whose input layer converts the block layout of any standard (H.264, HEVC, AV1) into one common image-like grid. It is trained on videos re-encoded under many settings, so it can be trained on one standard and deployed on another. The target setting is continuous camera monitoring of the cell cultures used to make therapeutic extracellular vesicles, where imaging volume is large and changing camera hardware is costly.

Extracellular vesicles, the nanoscale membrane packets that cells release and that are now being developed as drug carriers, are moving from mouse-scale flask culture to automated 3D bioreactors. Advancements in extracellular vesicles biomanufacturing surveys this shift and counts 151 formal clinical trials, and it names the lack of standardized in-line quality control as an open problem. The enclosed microcarrier system in Automated and Enclosed Three-Dimensional Biofabrication System points to the same gap. Continuous imaging is the obvious sensor for this. In a regulated plant, though, a monitoring model is validated against a specific data pipeline, and a vendor firmware update that switches the camera from H.264 to HEVC can quietly invalidate that model. Here, codec dependency is a validation hazard and not only an inconvenience. We treat it as the core technical problem, and biomanufacturing as the setting that makes it matter.

Core Claim

The claim has two parts. First, the loss in tracking accuracy when a compressed-domain tracker is moved to an unseen codec can be made small if the input exposes three things on a fixed-resolution pixel grid: the partition geometry (the size and shape of the block each pixel belongs to), the block motion, and the residual energy. Training then uses codec and bitrate re-encoding as augmentation. Second, at matched tracking accuracy, this tracker needs substantially less compute per frame than a decoded-pixel tracker, because it detects cells only on the standalone keyframes and propagates them through the in-between frames using the motion data the encoder already produced.

What is new is not compressed-domain tracking of cells. The file contains no such work, but that combination alone is thin. The new part is an explicit, tested claim about cross-codec transfer, which prior compressed-domain systems concede they lack. Beyond Pixels (STREAM) mines H.264 motion vectors and residuals for AI-generated-video detection and states it cannot ingest HEVC or AV1. Plug-and-Play Versatile Compressed Video Enhancement uses H.264 partition maps and motion vectors for restoration and is likewise H.264-only. Deformable Video Transformer frames the general trap: compressed-domain gains depend on the specific format being present. These works motivate the gap. We keep their core ingredients (motion-guided feature warping, residual gating) and replace the codec-bound input layer.

Research Program

The input representation, which we call the motion canvas, is built by a parser that performs entropy decoding of the bitstream but skips pixel reconstruction and the in-loop filters. For each inter-coded frame, the parser writes onto a grid at 1/4 of image resolution a set of per-cell quantities. Each cell records the block motion vector of the covering prediction unit and the log width and log height of that block, which captures H.264's fixed 16×16 macroblocks, HEVC's recursive quad-tree splits up to 64×64, and AV1's rectangular splits in one geometric description. Each cell also records the prediction mode, the energy of the quantized residual coefficients, and the frame's quantizer. The partition channels matter for a physical reason. Encoders split blocks finely where content is inhomogeneous, such as at cell boundaries, and use large blocks over flat medium, so block size acts as a free, codec-independent boundary and confidence cue.

The tracker detects cells or microcarriers only on intra-coded keyframes, using a small U-Net on the decoded keyframe. It then propagates candidate positions through the following predicted frames with a learned warping module over the motion canvas. This follows the back-tracing idea of Compressed Video Action Recognition, which accumulates motion back to the reference keyframe. A light correction head reads residual energy to flag candidates whose appearance has changed, such as a cell dividing, detaching, or entering the field. The propagated candidates, carrying position, area, and intensity features, go to the transformer association head of TRACKASTRA. That paper links shallow per-cell features across a sliding window and enforces at most one parent per cell with its "parental softmax". We reuse it unchanged, so any difference in results comes from the front end.

Training uses codec augmentation. Every raw training sequence is re-encoded with x264, x265, and libaom-AV1 at several quality settings and group-of-pictures lengths. The main transfer experiments hold out one codec entirely.

A physical caveat constrains the design. Block matching is built for smooth, rigid motion; STREAM even treats erratic motion-vector patterns as a sign of synthetic video. Cells deform and sub-micron particles diffuse randomly, so we expect motion vectors to be reliable for cells and microcarriers but close to noise for vesicle-scale particles. We therefore scope the main claim to cells and microcarriers, and treat particle-scale data only as a diagnostic.

Baselines

The naive reference point is the decoded-pixel pipeline. Each compressed video is fully decoded with a hardware decoder and passed to TRACKASTRA with its standard detector. On uncompressed data, TRACKASTRA reports an AOGM tracking error of 23 on BACTERIA, down from 118 for Delta 2.0, and 7.9 on DeepCell DynamicNuclearNet, down from 18.1 for Caliban. AOGM is a graph-edit count of tracking mistakes, where lower is better. This is the strongest existing method in the file with reported numbers. We will run it on exactly the same re-encoded videos, with the same detector training data and the same hyperparameter search budget as our model. Beating it means matching its AOGM within 10% while using at least 3× less end-to-end compute per frame, counting parsing and decoding time. A lower error alone is not required.

A second decoded-pixel reference is CELLECT from CELLECT: contrastive embedding learning for large-scale efficient cell tracking. It embeds cells with a 3D U-Net and reports 2 seconds per frame, 56× faster than linajea, with 46% long-trajectory accuracy on C. elegans. It is the strongest "scale by a more efficient pixel model" competitor, and we run it on the volumetric Cell Tracking Challenge sequences under the same compute accounting.

On CTMC, the file reports an average tracking accuracy (TRA) of 0.39 for a classic Viterbi tracker, 0.32 for DeepCell, and a multiple-object tracking accuracy (MOTA) of 23.59 for Tracktor with Faster R-CNN. These serve as floor references from CTMC.

The compressed-domain reference is an H.264-only variant of our tracker. Its input is macroblock motion vectors and residuals in the style of STREAM and CoViAR, with no partition channels and no augmentation, trained and tested per codec. This shows what codec-bound design already achieves in-distribution.

Two ablations isolate the key ingredient. One drops the partition-geometry channels. The other drops codec augmentation. If either matches the full model on held-out codecs, the claimed mechanism is wrong.

Why not just…?

Why not just decode everything and run a pixel tracker? This is the most serious threat. Hardware decoders are fast, and much of the speedup reported for CoViAR (4.6× over Res3D) and STREAM (2.5–11×) comes from smaller networks on sparse inputs rather than from skipping decoding itself. Our parser still entropy-decodes, which is a large share of decoding cost for AV1. The TRACKASTRA baseline, with full compute accounting, settles the question directly. If it reaches our accuracy at comparable cost, the efficiency half of the claim fails, and only the robustness findings survive.

Why not transcode every feed to H.264 at ingestion and use a codec-specific model? This removes the dependency by fiat. However, transcoding requires a full decode plus a re-encode, which costs more than the pixel baseline, and each re-encode passes through lossy loop filters again. STREAM argues this kind of processing erases high-frequency detail. We include a transcode-then-H.264-model arm with measured cost and accuracy. If it is cheap enough on commodity hardware, that is a legitimate practical answer, and we will report it as such.

Why not just track on keyframes at a low frame rate, since cell growth is slow? For confluence and density estimates over hours, this probably suffices, and we will not claim otherwise. It fails for identity-preserving tracking when displacements between samples are large; the crowded BACTERIA sequences in TRACKASTRA are exactly that regime. A keyframes-only arm, with interpolated linking at matched compute, will show at what keyframe spacing identity switches overtake our propagation.

Why not rasterize motion vectors into a dense flow field and skip the learned canvas? This is codec-agnostic by construction and is the cleanest competitor to our key ingredient. It appears as the partition-channel ablation run without augmentation, so the result will show whether geometry and training add anything.

Proposed Experiments

We re-encode raw sequences from the Cell Tracking Challenge (The Cell Tracking Challenge: 10 years of objective benchmarking), which offers 20 datasets with dense silver-standard annotations, and from CTMC, which offers 86 DIC videos across 14 cell lines. Microscopy data above 8 bits is encoded in 10-bit HEVC and AV1 profiles, and bit-depth effects are reported separately. A secondary set comes from brightfield imaging of MSC microcarrier culture. It will be collected in-house because the file lists no public bioreactor video benchmark, and it is used only for qualitative transfer.

The decisive check is leave-one-codec-out. We train on H.264 and HEVC and test on AV1, rotating which codec is held out. We report the ratio of held-out to in-distribution error, in AOGM and TRA, for the full model, both ablations, and the H.264-only reference. The claim holds if the full model keeps this ratio near 1 across quality settings while the ablations do not. The second check is an accuracy-versus-compute curve against the TRACKASTRA pixel pipeline, measured in joules and wall-clock time per frame on identical hardware.

A diagnostic third check, on the ISBI vesicle sequences, tests the decoding-damage question: does residual energy read before loop filtering detect sub-resolution particles better than the decoded pixels do? The file reports no result for those sequences (no reported result found on Lacuna for the ISBI vesicle tracking benchmark), so this experiment establishes its own baselines. We also expect the motion channels to fail there, as argued above.