Many modern image-text models are trained so that a picture and its caption land near each other in a shared embedding space. To make that training cheaper, researchers use dataset distillation: they compress tens of thousands of real image-caption pairs into about a hundred synthetic pairs that train a model almost as well. The fastest method so far, Efficient Multi-modal Dataset Distillation via Analytic Parameter Matching (APM), uses a shortcut. It solves in closed form for the small projection layers that map frozen image and text features into the shared space. It then optimizes the synthetic data so that the closed-form solution computed on the synthetic data matches the one computed on the real data. This removes the long training replays that older methods need.
APM assumes each image has exactly one caption. Real data rarely looks like that. MS-COCO has five captions per image. In physical sensing, one tactile imprint can come from many visual states and one visual state can produce many imprints. We propose to rewrite the closed-form target so that it is computed from the full correspondence graph, meaning the record of which items in one modality belong with which items in the other. The synthetic set then becomes a small bipartite graph with its own learnable links, not a list of pairs. It may also hold different numbers of images and captions. The practical claim is that the one-to-many structure can be distilled once and exactly, without the "sub-dataset cycling" heuristic APM uses on COCO, and without paying for duplicated images in the synthetic budget.
Core Claim
Closed-form projector solutions are built from summary statistics of the training pairs. These are the covariance of image features, the covariance of text features, and the cross-covariance between them. Every such statistic can be rewritten as a sum weighted by a correspondence matrix, not a sum over matched pairs. Let $X \in \mathbb{R}^{N \times d}$ be frozen image features, $T \in \mathbb{R}^{M \times d}$ frozen text features, and $C \in \mathbb{R}^{N \times M}$ a nonnegative correspondence matrix normalized to total mass 1. Let $D_x = \mathrm{diag}(C\mathbf{1})$ and $D_t = \mathrm{diag}(C^\top \mathbf{1})$ be its marginals. The graph-weighted statistics are
$$ \Sigma_{xx} = X^\top D_x X,\qquad \Sigma_{tt} = T^\top D_t T,\qquad \Sigma_{xt} = X^\top C\, T . $$
APM's one-to-one case is the special case where $C$ is a scaled identity. The teacher target is computed once from $(X, T, C)$ on the full dataset. This covers COCO's five captions, soft temporal windows in sensor streams, or any other correspondence structure. The student is a synthetic triple $(X_s, T_s, C_s)$ with $n$ images, $m$ texts and a learnable $n \times m$ correspondence matrix. It is optimized so that its analytic projector parameters match the teacher's.
There is a subtlety that makes the claim testable and not obvious. For a plain least-squares projector from images to text, repeating each image once per caption gives exactly the same solution as regressing onto the mean caption. So in that direction, averaging captions loses nothing. The text-to-image direction is different: its covariance term $\Sigma_{tt}$ includes the spread among captions of the same image. Averaging throws that spread away, and a synthetic set with one text per image cannot represent it. We therefore predict three things. Gains should appear mainly in text-to-image behavior and in text retrieval robustness. They should grow with how diverse the captions are. They should vanish on data with truly one-to-one pairing. If APM's InfoNCE-derived solution does not decompose into these moments as cleanly as ridge regression, the first task is to find which of its terms carry the correspondence sums. Our method generalizes exactly those terms.
Research Program
Derivation. The first piece of work re-derives APM's analytic parameters, including its stabilizations for embedding shift, scale explosion and near-singular matrix inversion, in the graph-weighted form above. We will verify that the result reduces exactly to APM when $C$ is the identity. This is a correctness check. The re-implementation must reproduce APM's Flickr30k result before anything else is claimed.
Learning the synthetic graph. The second piece makes $C_s$ learnable. We parameterize it by a softmax over a small score matrix, with an entropy penalty so it does not collapse to a single dense block. We initialize the synthetic nodes with the prototype-mining idea from Asynchronous Matching with Dynamic Sampling for Multimodal Dataset Distillation (AMD), which runs K-means on joint embeddings, extended to cluster images and captions separately. AMD also motivates the third piece. It shows that text representations settle early during distillation while image representations keep changing. That suggests the best synthetic set may be lopsided, for example fewer images each linked to several captions. We will sweep the ratio $n:m$ at a fixed budget.
Physical sensing. The fourth piece carries the method into the regime where many-to-many correspondence is structural, not an annotation artifact. Learning to Jointly Understand Visual and Tactile Signals builds a shared vision-touch space and notes that grasps and pressures map many-to-many. TactileVAD calls the same effect geometric aliasing: distinct physical states produce identical tactile imprints. Both papers motivate the extension, and neither distills anything.
We will use BAAI-Humanoid/DECO-50 for this. It contains synchronized camera frames and 1,062-point tactile arrays from bimanual manipulation. The procedure has three steps. First, train frozen vision and tactile encoders contrastively on the full data, in the style of Geometric Multimodal Contrastive Representation Learning (GMC). Second, define $C$ with a temporal kernel, so a tactile reading links softly to nearby frames. Third, distill a small graph whose analytic projectors reproduce the teacher shared space. The question is whether a few hundred synthetic nodes can stand in for millions of frames when fitting the alignment layers.
This is deliberately narrower than distilling a whole manipulation policy. The file notes that policies like DECO use diffusion objectives with no known closed form, so we do not attempt that.
Baselines
Naive references. The do-nothing reference is training the projectors on a random real subset of the same size. For COCO, the subset keeps all five captions of each sampled image, so the reference itself uses the one-to-many structure. We also include Herding, which AMD reports is clearly beaten by distillation on Flickr30k. We still need it to show the gap persists under our budget accounting.
APM. The strongest existing method is APM, run in two forms. The first is its published sub-dataset cycling for COCO. The second is a "flattened" variant that treats every caption as its own pair and duplicates the image. The paper reports 12.8% image retrieval at rank 1 on Flickr30k with 100 synthetic pairs. That beats RepBlend's 11.5% and LoRS's 8.3%, at 1.2 seconds per iteration and 0.5 GB of offline storage against LoRS's 11.5 seconds and 32.6 GB. We found no reported result on Lacuna for APM's COCO or AudioCaps retrieval numbers, so we will produce those ourselves with the authors' protocol. All methods share the same frozen encoders, the same synthetic budget, the same iteration count, and an equal hyperparameter search of 20 trials each on a held-out split.
Budget accounting. Our synthetic sets may hold different numbers of images and texts, so a fair budget matters. We report results at equal item count and at equal stored bytes. Images cost more than captions, so equal bytes is the stricter test.
Trajectory-matching methods. We include Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation (RepBlend) and AMD as the strongest methods of that type. AMD is reported at +4.5 points image retrieval at rank 1 over LoRS on Flickr30k at 200 pairs. We run them at matched wall-clock time to show where accuracy trades against cost.
Mean-caption APM. This is the most important competitor. APM is run on the per-image mean caption embedding. For reasons given above, it should equal our method on image-to-text statistics.
Ablations. Two ablations isolate the key ingredient. One uses the graph-weighted teacher target but a fixed identity $C_s$ for the synthetic set; it tests whether a learnable synthetic graph matters or only the corrected target does. The other forces $n = m$; it tests whether lopsided budgets matter.
Beating the baselines. Beating APM means at least 1 point higher mean recall on COCO and on DECO-50 vision-tactile retrieval, at equal byte budget, across three seeds. It also means no loss on Flickr30k, where the method must reduce to APM. The cost of a distillation iteration must stay within 1.5 times APM's. Beating mean-caption APM specifically requires a gain in text-to-image retrieval.
Why not just…?
Why not just average the captions and run APM unchanged? This is the real threat. For the image-to-text direction it is provably as good. Our answer rests only on the text side, where averaging deletes the caption covariance that the text-to-image projector depends on. The mean-caption baseline will settle this directly. We will also run a controlled test: subsample COCO to one to five captions per image, and separately construct caption sets of low and high diversity. If mean-caption APM matches us at every diversity level, the correspondence-graph machinery is unnecessary for COCO-style data. The proposal would then shrink to the vision-touch setting, where no natural "mean" partner exists under temporal many-to-many links.
Why not just treat each caption as a separate pair, as flattened APM does? Flattening is exact on the teacher side. It is wasteful on the student side: to express a one-to-many link it must spend synthetic budget on duplicated images. The equal-byte comparison against flattened APM, and the $n:m$ sweep, measure exactly this waste.
Why not just keep a small real coreset with all its captions, or generate more real-looking data with foundation models? AMD and RepBlend already show that distilled sets beat Random and Herding subsets at 100–200 pairs on Flickr30k. We rerun that comparison under one-to-many accounting and do not assume it carries over.
Generating more data with foundation models is the expensive path documented by TrueTeacher and InstructDET. TrueTeacher used a 540-billion-parameter labeler. InstructDET used a four-model generation and filtering cascade. For tactile data, no such generator is readily available, and DECO-50 itself took 50 hours of teleoperation. Neither approach reduces the cost of fitting the alignment layers, which is the target here.
Proposed Experiments
Correctness check. On Flickr30k at 100 and 200 synthetic pairs, where $C$ is effectively one-to-one, our implementation must reproduce APM's 12.8% image retrieval at rank 1 within seed noise. Failing this invalidates everything downstream.
Main test (MS-COCO). We compare the graph-weighted method, with learned $C_s$ and lopsided budgets, against sub-dataset-cycling APM, flattened APM, mean-caption APM, RepBlend, AMD, Random and Herding. Budgets are 100, 200 and 500 at equal item count and equal bytes. We report image and text retrieval at ranks 1, 5 and 10, seconds per iteration, peak memory and offline storage.
Mechanism check. We rerun the diversity-controlled COCO subsamples described above. The prediction is that the gap over mean-caption APM grows with caption diversity and appears mainly in text-to-image retrieval. A flat gap would indicate the gains come from something other than the correspondence covariance.
Cross-domain replication (AudioCaps). AudioCaps also has multiple captions per clip, so the COCO findings should repeat there.
Physical extension (DECO-50). We measure vision-to-tactile and tactile-to-vision retrieval, plus a missing-modality probe in the spirit of GMC's sound-only evaluation. The probe asks whether projectors fitted on distilled data still support object and task classification from vision alone. We compare against a random real subset of synchronized pairs and against APM applied to hard-thresholded one-to-one pairs. We found no reported result on Lacuna for retrieval on DECO-50, so the full-data projector trained on the same frozen encoders sets the ceiling.
Decisive outcome. The decisive result is a positive COCO margin over mean-caption APM that scales with caption diversity, together with a DECO-50 margin over one-to-one APM. Missing either one would locate precisely where closed-form distillation needs the full correspondence graph and where a simple average suffices.
References
- Efficient Multi-modal Dataset Distillation via Analytic Parameter Matching
- Asynchronous Matching with Dynamic Sampling for Multimodal Dataset Distillation
- Learning to Jointly Understand Visual and Tactile Signals
- TactileVAD: Geometric Aliasing-Aware Dynamics for High-Resolution Tactile Control
- Geometric Multimodal Contrastive Representation Learning
- Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation
- TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
- InstructDET: Diversifying Referring Object Detection with Generalized Instructions