Touch as a Cheap Assay: Visuo-Tactile Signals from Routine Handling in Formulation Search

Self-driving chemistry labs are robots paired with an optimizer that pick the next experiment. They spend most of their budget on slow, expensive endpoint measurements. Meanwhile the robot's own hands, which tilt, swirl, stir and pour every sample, throw away physical information about the material they are touching. A vial of a thickened surfactant solution resists a stirring rod differently from water. It lags when tilted, and it pulls on a gripper's soft fingertip in a telltale way. We propose treating these handling signals as a cheap, noisy measurement of the property being optimized, and folding them into the optimizer as a second, low-cost information source. The optimizer then calls the expensive instrument only when touch alone cannot settle the question.

The concrete claim is testable. In a formulation space where the target is mechanical, such as viscosity or gelation, adding fused camera-plus-touch estimates from ordinary handling motions should cut the number of expensive reference measurements needed to find a formulation in a target window. Touch should also let the system discard obvious failures before they reach the reference instrument. The same experiment tells us which part of the sensing matters. It compares full tactile fusion against a single wrist force sensor and against vision alone, so the answer to "is touch worth it?" is measured rather than assumed.

Why this gap is real

Existing self-driving labs measure outcomes optically or with offline assays. RAISE runs Bayesian optimization, a search method that fits a probabilistic model of the objective and picks experiments that balance promise and uncertainty. It uses Thompson sampling over surfactant mixtures and scores each mixture by fitting droplet shapes in backlit images. The agentic SEDDS system pairs a proposer and a reviewer LLM with a robotic dispensing and spectrophotometric assay. Its results arrive in 24-hour batches. Both systems treat execution as a black box that returns one scalar hours later.

On the robotics side, ORGANA and Chemistry Lab Automation via Constrained Task and Motion Planning automate general-purpose lab manipulation. They rely on visual turbidity proxies for measurement. ORGANA explicitly names haptic feedback as missing future work, and neither system connects to an optimizer that explores chemical space. MATTERIX simulates lab manipulation and reaction kinetics but models no tactile contact.

These papers motivate the proposal. They show the loop from physical interaction to search is open. None of them uses contact data as evidence about the chemistry.

Visuo-tactile work supplies the sensing and fusion methods and also constrains how the fusion is built. DECO finds that naively combining camera and tactile inputs lets the high-bandwidth camera stream drown out sparse touch cues. Its fix is to inject touch through a separate lightweight cross-attention adapter, and we borrow this decoupled design for our property estimator. FAVLA shows that force signals at roughly 1 kHz must be processed at their own rate rather than resampled to camera speed. We therefore summarize the fast force and tactile stream with a dedicated temporal encoder. DIFFTACTILE simulates soft-sensor contact with deformable and elastoplastic materials using finite-element and material-point methods. We use it to build a simulated version of the task in which the true material parameters are known.

What differs from all of these is the role touch plays. The robotics papers use touch to make a manipulation succeed. We use it to measure the sample.

Core Claim

Our hypothesis concerns formulations whose target property is mechanical. For these, a short scripted handling motion lasting a few seconds yields a fused visuo-tactile estimate that correlates well with a reference measurement from a rheometer, an instrument that measures how a fluid flows and deforms. An optimizer that models the tactile estimate and the reference measurement jointly, and chooses which one to buy for each candidate, should reach the target with fewer reference calls than an optimizer that sees only reference measurements. The model we use for this is a multi-fidelity Gaussian process: a single model over both measurement sources that learns how closely they track each other.

The optimizer spends its budget with a cost-aware rule. For a candidate formulation $x$ and measurement source $s$ (tactile or reference), it picks the pair that maximizes

$$ \alpha(x, s) = \frac{I\big(y^{*};\, y_s(x) \mid \mathcal{D}\big)}{c_s}, $$

The terms are as follows. $I$ is the expected reduction in uncertainty about the best achievable objective value $y^{*}$ from observing source $s$ at $x$. $\mathcal{D}$ is the data collected so far. $c_s$ is the cost of that source, measured in robot-minutes plus consumables.

There is one additional behavior. When the tactile estimate places a candidate confidently outside the target window, the system never sends it to the reference instrument. An example is a sample that has already gelled when the goal is a pourable liquid. This early pruning is how fast physical signals reach the slow search, without asking a chemistry reasoner to run at the robot's control rate.

Research Program

The work proceeds in three stages. Each stage can fail informatively.

The first stage builds the tactile property estimator. A robot arm carries a 6-axis wrist force/torque sensor and optical tactile fingertips of the GelSight type, which image the deformation of a soft gel pad. The arm grasps capped vials and runs three scripted motions: tilt-and-hold, swirl, and a slow stir with a rod.

The estimator has two parts. A temporal encoder processes the high-rate force and tactile streams at their native rate. A camera encoder watches the free surface lag behind the vial wall. Following DECO's lesson, the touch stream enters through its own cross-attention adapter rather than being concatenated with image features. The output is a predicted log-viscosity together with a predicted variance.

We train the estimator on calibration standards whose properties are known independently of the chemistry under study, such as glycerol–water mixtures and commercial viscosity standards. We then test whether it transfers to the actual formulation chemistry.

The second stage builds a simulated counterpart. We use DIFFTACTILE's material-point simulation, with material parameters driven by a known synthetic function of composition. There the true optimum is known, which lets us isolate the search logic from sensor error.

The third stage is the closed loop on hardware. The formulation space consists of aqueous surfactant–salt–polymer mixtures. SDS, the surfactant RAISE uses, forms structured phases whose viscosity rises sharply with salt concentration. This gives a rugged, cliff-like landscape that mirrors activity cliffs in molecular search. The objective is to land in a narrow viscosity window while also meeting a contact-angle target measured optically as in RAISE. This makes the task genuinely multimodal.

Baselines

The naive reference point is uniform random sampling of the formulation space. Every sample is scored with the rheometer, under the same total cost budget. This sets the floor that any method must clear.

The strongest existing method for this question is reference-only Bayesian optimization of the kind RAISE runs. We use the BayBe library with Thompson sampling over the same composition variables and the same rheometer-plus-contact-angle objective, and it gets the identical cost budget and the same initial design. RAISE reports locating target formulations "within the first 4% of the total possible experimental space" (RAISE). That figure was measured on a contact-angle objective, and we found no reported result on Lacuna for viscosity-targeted formulation search. The comparison therefore has to be run fresh here. Beating this baseline means reaching the viscosity window with at least 30% fewer rheometer calls at equal total cost. We also require a lower median simple regret, the gap between the best objective value found and the true best, after a fixed budget, across at least five independent campaigns with different random initial designs.

The second strong competitor is the agentic proposer–reviewer approach from the SEDDS work. It reached in 16 experiments a target that gradient-boosted Bayesian optimization needed 64 to reach. We run it with the same LLM access, the same reference data, and the same experiment budget. It is given a text summary of each handling trial, such as "stir torque peaked at a stated value; surface lag of a stated duration," as a strong but simple way to put touch into the loop.

Two ablations isolate the key ingredient. The first keeps the multi-fidelity optimizer but replaces the fused estimator with single-source estimators: wrist force/torque only, vision only (video of the tilting vial), and tactile fingertips only. The second keeps the fused estimator but disables either early pruning or the cost-aware choice of measurement source.

Why not just…?

Why not just buy a rheometer and run it on everything? The rheometer is the ground truth, and we keep it. The question is whether it must be called on every candidate. RAISE's value came from throughput of roughly one contact angle per minute, compared with slow manual goniometry. A rheology measurement costs loading, running and cleaning time. If our cost-aware loop does not reduce reference calls against reference-only BO at equal total cost, the idea fails, and the head-to-head comparison is exactly that test.

Why not just use a wrist force sensor or stirrer motor current instead of tactile skins? This alternative is a real threat. Stir torque is a classic, simple viscosity proxy, and it may carry most of the useful signal. The single-source ablation settles the question directly. If force/torque alone matches fusion in estimator correlation and in the number of reference calls saved, we report that. The contribution then becomes the handling-as-assay loop, with the cheapest sensor that works. We expect tactile skins to matter most where torque is uninformative: inside a sealed vial with no rod, and for adhesive or elastic phases where shear at the fingertip, rather than bulk torque, reveals the material's state. That expectation is a hypothesis and needs no concession in advance.

Why not just use vision, watching the liquid move when tilted? Vision is cheap and non-contact. However, the corpus documents that it fails in exactly this setting. ORGANA needed dedicated models to perceive transparent glassware at all. The visuo-tactile survey in the file (Visuotactile Fusion for Contact-Rich Robotic Manipulation) notes that vision degrades with transparent and reflective media and cannot measure compliance directly. A clear gel and a clear low-viscosity solution can look identical until they move, and even then only the free surface is visible. The vision-only ablation measures this directly on clear formulations.

Why not just give touch readings to an LLM agent as text? This is cheap, and the SEDDS results show that in-context lab evidence is powerful: removing historical data dropped agent performance by 45% (Agentic Systems for Sample-Efficient Drug Formulation Design). We therefore include it as a baseline rather than dismissing it. Its weakness is calibration, which BoMolLLM also identifies for semantic surrogates. It has no principled way to decide when a noisy tactile summary is trustworthy enough to skip the reference instrument. We will measure how often each method prunes a candidate that was in fact inside the target window.

Proposed Experiments

The first decisive check happens before any optimization. On held-out formulations from the target chemistry, the fused estimator must reach a rank correlation above 0.8 with rheometer log-viscosity. Its predicted uncertainty must be calibrated, meaning the fraction of true values inside its 90% intervals should be close to 90%. If the estimator fails this on the real chemistry after being trained on glycerol standards, the loop cannot help. That failure is itself an informative result about how well tactile rheology transfers across materials.

The second check runs in simulation. In the DIFFTACTILE-based benchmark with a known optimum, we plot simple regret against cumulative cost for all baselines. We vary how noisy the simulated tactile source is, to find the correlation level below which the cheap source stops paying for itself. Multi-fidelity reasoning predicts that the saving shrinks as this correlation falls. The hardware result should land where the measured correlation from the first check says it will.

The third check is the hardware campaign. Five campaigns per method are run on the surfactant–salt–polymer space. We report the rheometer calls needed to first reach the viscosity-and-contact-angle target, the regret at a fixed budget, and the rate of false prunes. As secondary outcomes, we log grasp slips and dropped vials with and without tactile fingertips, because touch should also make the handling itself more reliable, as ORGANA anticipated.

The proposal succeeds if fusion-driven multi-fidelity search beats reference-only BO on reference calls, with a false-prune rate under 5%. It is informative in a different way if a single force sensor does just as well.