Physical Tracebacks: Refining Optoelectronic-Tweezer Control Code Without Spending the Cells

Optoelectronic tweezers are a way to move microscopic objects, such as cells, beads, and droplets, by shining a projected image onto a light-sensitive chip. Wherever light falls, the chip briefly acts like a metal electrode, and the resulting electric field pushes or pulls nearby particles. The underlying effect is called optically-induced dielectrophoresis. Because the "electrodes" are just pixels from a projector, the control program is ordinary software: it decides which shapes to draw, where to move them, and at what AC frequency and voltage to drive the chip. Large language models (LLMs) are now good at writing such software, and they get much better when they can run their code, read what went wrong, and patch it. That loop is called iterative code refinement. On a tweezer chip, though, "running the code" means exposing living cells to light and electric fields. Every failed attempt heats the sample and can kill it, and the simulators that could stand in for the chip are known to be off by large factors.

We propose to make iterative refinement work for tweezer control code by changing two things. The first is what the model is told after a failed run. We convert microscope tracking into typed physical tracebacks, short structured failure reports such as "particle 3 lagged its trap by 14 µm at 40 µm/s, which exceeds the measured stiffness limit" or "cage 2 shrank below the 22 µm escape threshold at frame 118." The model reads these instead of raw coordinate logs. The second is where the failures happen. Refinement iterations are spent on inexpensive stand-ins, namely polystyrene beads in the same buffer plus a simulator re-fitted after every bead run. The live-cell run then gets a small, fixed number of attempts with automatic rollback to the last safe light pattern. The claim is narrow and testable. On tweezer tasks that need new control structure, and not just new parameter values, this loop should reach higher success at a fraction of the light and field exposure per cell, compared with refinement on raw logs or black-box parameter search.

What exists and where the gap is

On the tweezer side, the strongest automated system is Zheng et al.. They combine YOLOv7 detection, Hungarian target assignment, Conflict-Based Search path planning, and a hand-written rule that keeps each light cage's dark interior at least 22 µm wide. With this they report a 100% trapping success rate while moving 10 beads collision-free through obstacle channels at 10 µm/s. There is no generated code in their system. When the physics departs from the rules, for example with irregular cells, heating, or a need to switch operating modes, nothing rewrites the controller. The closest LLM work is Jia et al., who fine-tune Llama 3-8B to translate instructions into electrode sequences on a hard-wired electrowetting array. That system is open-loop, and it does not use optical virtual electrodes at all. Liang et al. learn adaptive droplet routing with reinforcement learning on discrete electrode grids. Their policy is not readable code, and it acts only in four directions. These three papers motivate the proposal: automation exists, but not closed-loop code refinement on a continuous optical-field device.

On the refinement side, the method comes from several papers. OpenCodeInterpreter shows that execution feedback raises HumanEval Pass@1 from 79.3% to 82.9%. Richer natural-language feedback raises it to 88.4%. That gap suggests the form of feedback matters more than its mere presence, and it is the lever we pull with typed tracebacks. RobotGPT refines robot scripts in a PyBullet simulator, with up to three corrections, and reaches 91.5% versus 38.5% for zero-shot generation. It then fails on real hardware because the physical run was open-loop and the simulator missed friction and placement error. MEIC adds a scorer agent and rollback to the last stable version, which lifts functional Verilog fixes to 78% against 66% for retrieval or fine-tuning baselines. We reuse the rollback idea directly.

Three papers constrain the design. Yan et al. (Yan et al.) find that refinement gains plateau after two to three turns. Neale et al. needed a post-hoc factor-of-4 reduction to make finite-element trap forces match measured forces on HeLa cells. Valley et al. and infrared thermography work document heating and conductivity limits on how long cells can be exposed. Taken together, these rule out two designs: long refinement on the cells themselves, and trusting an uncalibrated simulator as the verifier. Our design follows from those two exclusions.

Core Claim

Refinement of physical actuation code fails in two ways. When feedback is a raw trajectory, the model has to infer the physics itself. When the verifier is either expensive (the live sample) or wrong (the uncalibrated simulator), iterations are either unaffordable or misleading. We address the first by mapping tracking data onto a small set of known tweezer failure modes, each with a measured threshold. We address the second by moving exploration to bead runs, which also recalibrate the simulator. The proposal is new in treating the physical experiment as a verifier whose output must be compiled into a traceback, and whose runs carry a budget that differs by sample type.

Research Program

The hardware is a standard tweezer setup of the kind in Zheng et al.: a projector based on a digital micromirror device, an amorphous-silicon photoconductive chip, and a camera. The chip also carries the insulating and fluoropolymer layers of Valley et al., so that frequencies below about 100 kHz move droplets and frequencies above it move particles. The LLM writes Python against a thin API that exposes drawing light shapes, setting their trajectories, setting AC frequency and voltage, and reading detections.

The traceback compiler is the main component. It runs the detector and tracker on every run and checks a fixed list of failure signatures. Trap lag is measured against the stiffness limit. The cage interior is checked against the 22 µm escape threshold. Defocus or sudden disappearance is used as the sign of vertical ejection. Drift of untrapped particles is used as the sign of heating-driven flow. The compiler also flags collisions and identity swaps. Each flag is reported with its frame, object, measured value, and threshold. Stiffness is re-estimated on each run from steady transport. At constant velocity, Stokes drag balances the restoring force, so

$$k = \frac{6\pi \eta r v}{\Delta x},$$

where $k$ is trap stiffness, $\eta$ is the medium viscosity, $r$ is particle radius, $v$ is transport speed, and $\Delta x$ is the steady lag behind the trap center. This relation holds because Neale et al. found the restoring force to be linear in displacement. The fitted $k$ does two jobs. It sets the lag threshold in future tracebacks, and it replaces the fixed factor of 4 as the correction applied to the simulator.

The refinement schedule has two stages. In the bead stage, the agent refines freely against beads sized to the target cells, alternating simulator runs with chip runs, up to a fixed number of chip runs. In the cell stage, the agent gets at most two attempts on live cells. A scorer, as in MEIC, watches every step of the run. If a traceback fires during a cell run, the controller rolls back to the last light pattern that held the cell stably, and the run stops rather than continuing to expose the cell. We log exposure as integrated illumination-time multiplied by applied field for each cell.

The task suite is built so that some tasks need new structure. Multi-bead navigation replicates Zheng et al. Size-based sorting of a bead mixture requires choosing frequencies that give opposite force signs. Concentration followed by droplet encapsulation requires switching between droplet mode and particle mode mid-protocol, as in Valley et al. Isolation of a single HeLa cell from a mixed population, followed by encapsulation, adds biological variation in size and electrical response. Each task is specified in natural language together with a programmatic success check.

Baselines

The naive reference is one-shot generation. The same LLM writes the controller once from the task description, in the style of Jia et al., and runs it without feedback. Any closed-loop method that does not beat it clearly is not worth its cost.

The strongest existing method on navigation is the classical pipeline of Zheng et al., which reports 100% trapping success with 10 beads at 10 µm/s. We will reimplement it on our chip with the same detector and obstacles. On that task we do not expect to beat it. It is the ceiling we aim to match while generating code. On sorting and encapsulation it has no native mechanism, so we give it a hand-engineered extension written by a human with the same wall-clock budget as the agent. Beating it means higher success on the structural tasks without losing more than a few points on navigation. There is no reported result found on Lacuna for any LLM-based controller on optoelectronic tweezers, so all agent numbers will be new.

The strongest refinement baseline is raw-log refinement in the style of RobotGPT and OpenCodeInterpreter. After each run the model receives tracked coordinates, detection counts, and any Python errors, and it gets the same number of chip runs, cell attempts, LLM calls, and token budget as our method. RobotGPT reports 91.5% success in simulation with this style of loop. Our method beats it if it achieves higher cell-stage success at equal exposure, or equal success at lower exposure, across repeated trials with confidence intervals.

Three ablations isolate our ingredients. The first swaps typed tracebacks for raw logs while keeping everything else fixed. The second removes the bead stage, spending the same total chip runs directly on cells. The third removes the per-run stiffness recalibration, using the uncorrected simulator or Neale's fixed factor instead. Each ablation should cost measurable success or exposure. If one does not, that ingredient is unnecessary.

Why not just…?

Why not just use Conflict-Based Search with geometric rules, as Zheng et al. do? On navigation it is probably as good as or better than any LLM loop. It is deterministic, and it reports 100% trapping success. We therefore treat navigation as a control task where we expect parity at best. Our claim concerns tasks where the controller must change its structure, such as frequency switching between droplet and particle modes or handling cell heterogeneity. There the rule set has to be rewritten by a person. If a human-written extension, given matched time, equals the agent on sorting and encapsulation, the proposal's value reduces to engineering convenience, and we will say so.

Why not just run Bayesian optimization over the parameters of a fixed controller? This is the most serious threat. If the tasks reduce to tuning trap size, speed, voltage, and frequency, a black-box optimizer needs no LLM and no traceback compiler. We will run it with the same chip-run budget over a parameterized version of each task controller written by an expert. If it matches the agent on all four tasks, then the structural-change hypothesis is false for these tasks, and we would redirect the work toward traceback-guided parameter search. The decisive evidence is the encapsulation task, where a fixed parameterization cannot express the needed ordering of modes unless someone hand-codes it in advance.

Why not just train a reinforcement-learning policy, as Liang et al. did for droplet grids? Their policy was trained in a simulated grid environment. For tweezers, that requires a trustworthy simulator, and Neale's factor-of-4 discrepancy says we do not have one. Training on the chip would take far more runs than a cell stage can afford. We will include a policy trained on the recalibrated simulator as a secondary baseline. Its sim-to-chip drop, set against the agent's, directly tests whether readable code plus tracebacks transfers better than a learned policy.

Why not just refine directly on the cells with more iterations? The plateau reported by Yan et al. suggests extra turns buy little. The heating documented by Valley et al. and thermography says they cost cell health. The second ablation measures this trade directly, reporting success against viability by live/dead staining.

Decisive Checks

The proposal stands or falls on three results. First, typed tracebacks must beat raw logs at equal budget on at least the sorting and encapsulation tasks. Second, bead-stage refinement must transfer. Concretely, cell-stage success within two attempts should approach bead-stage success, while cell exposure falls well below that of direct cell refinement at comparable viability. Third, the agent must beat Bayesian optimization on encapsulation. For a sanity anchor we report concentration efficiency on the encapsulation task against the 93% that Valley et al. achieved by hand. If typed tracebacks help but Bayesian optimization matches the agent, the contribution shrinks to a feedback format and a calibration procedure for physical-device automation. That would still be useful, but it would be a smaller result than the one proposed here.