A robot that grips a ripe tomato needs more than the noun. The modifier also matters: "gently," "firmly," "barely." Recent work shows that a robot policy with a camera-based touch sensor can turn such words into measurably different grip forces. These sensors are elastomer pads watched by an internal camera, which reads contact force from how the gel deforms. Tabero reports that "gentle" instructions cut average grip force by over 70% relative to "firm" ones. But every such system is English-only. The usual ways of extending it to other languages are to translate the instruction first or to swap in a multilingual text encoder. Both are judged on text: did the translation look right, are the embeddings close? Neither measures what actually matters, which is whether the robot squeezed the right amount.
We propose to judge cross-lingual instruction following for force-sensitive manipulation the way code translation is judged: by execution. Two instructions in different languages count as equivalent if they make the robot apply the same force profile and achieve the same outcome, as read out by the touch sensor and the simulator. We build a small, standardized execution benchmark of graded force words in a high-resource donor language and in related low-resource languages. We then test a specific hypothesis. Intensity gradations are often expressed through reduplication, intensifier particles, or expressive word forms that translation tends to collapse. These gradations transfer better when the policy's force head is aligned through a typologically related donor language, using executed force as the training target, than when the instruction is routed through English.
Core Claim
The claim has two parts, and the first stands even if the second fails.
First, text-level agreement and physical agreement come apart for force-modulating instructions. Translation quality scores and embedding similarity will misrank systems relative to measured force. This mirrors what XCodeEval found for code, where lexical similarity predicted functional correctness poorly. The reason is specific to this setting. A force word's meaning is a position on an ordered scale. A translation can keep the word's general sense ("softly") while losing its rank relative to its neighbors, and only execution reveals the lost rank.
Second, aligning a low-resource language's force words to the policy through a related donor language preserves that ordering better than aligning through English. The alignment target is the force profile measured when the donor-language instruction is executed, not the donor sentence's embedding. This prediction comes from NLP findings that related donor languages beat distant high-resource ones: Role of Language Relatedness in Multilingual Fine-tuning shows gains within Indo-Aryan and harm from distant languages, and Multilingual Argument Mining shows family-specific subsets matching or beating large multilingual mixes. Neither of those works has an execution signal, and no tactile policy has been tested in a second language.
Positioning
Three lines of prior work supply the pieces, and none of them contains the combination proposed here.
Tabero is the starting system and the motivating evidence. Its high-level vision-tactile-language-action model outputs a target pose and a target force. A 500 Hz controller then regulates that force using marker-displacement tactile tokens, and the model generalizes zero-shot to the unseen English adverb "lightly." It is monolingual and simulation-only. Its ultra-gentle failure mode is central to our design: at roughly 10% force, objects slip, so a correct force setpoint does not guarantee a correct execution.
CTSRL aligns touch with frozen English CLIP text embeddings and frames sensor transfer as "translation for touch." It is evaluated only offline.
CL2CM transfers from donor languages to improve image-text retrieval in low-resource languages. It is passive retrieval, with no actions and no physics.
Code translation supplies the evaluation logic and part of the training method. TransCoder judges translated programs by matching reference executions on shared inputs. CoTran trains on compiler and symbolic-execution feedback and gains +14.89 functional-equivalence points over supervised fine-tuning for Python-to-Java. We borrow the notion of equivalence-by-execution. We do not borrow the reinforcement learning, for reasons given below.
Standardized evaluation work constrains the benchmark design. TacEva standardizes force calibration across sensor types and explicitly leaves task-level metrics to future work. It also finds that marker patterns help shear estimation but hurt localization, and that one sensor's error grew tenfold under some lighting. BARS and NerfBaselines show that unshared harnesses manufacture gains. We therefore fix the force readout, the tasks, the seeds, and the tuning budgets for every system.
Research Program
Benchmark. We extend Tabero's Isaac Lab tasks, which are LIBERO manipulations with simulated GelSight sensors, with an instruction set of graded force expressions in five tiers from "barely touching" to "very firmly." The language groups are:
- Hindi as donor, with Bhojpuri and Maithili as related low-resource targets.
- Indonesian as donor, with Javanese and Sundanese as targets. This follows Fajri Koto's finding that national pivot languages bridge regional ones.
- English throughout as the distant donor.
Ground-truth force scale. The ground truth for each expression comes from people, not from translation. Native speakers press on a handheld force gauge to enact each phrase in a fixed pinch. The median force per phrase, normalized to each speaker's "firm," defines a target intensity. The between-speaker spread defines a tolerance. This yields a per-language force scale that does not assume English adverbs are the reference.
Two force readouts. Executed force is recorded two ways: the simulator's true contact force, and the force estimated from the tactile image by a regressor calibrated with a TacEva-style indentation protocol. We also run each condition on a marker and a marker-less simulated sensor. A cross-lingual finding should survive both readouts. If the conclusion flips between sensors, the sensor is the confound, not the language.
Metrics. For each instruction we record three numbers from execution. Task success is whether the object is lifted and placed without dropping. Tabero's average applied force and maximum transient grip force are recorded as well. From these we derive three scores:
- Execution-equivalence rate: the fraction of target-language instructions whose executed mean force falls within the human tolerance of their elicited target.
- Ordering fidelity: Kendall's τ between the elicited intensity order and the executed force order within each language.
- Safe-gentle success: success rate restricted to the two gentlest tiers, where Tabero's slip failure lives.
Method. The policy's language encoder is a multilingual one, initialized from the English-trained Tabero-VTLA. We fine-tune only a small adapter so that a target-language instruction produces the same force setpoint the policy produces for the donor-language instruction of matching elicited intensity.
The execution component is a second training stage. Rollouts in which the setpoint is right but the executed force or success diverges, such as slips at low force, are relabeled with the setpoint that achieved the elicited force in donor-language rollouts. The adapter is refit on these corrected pairs. This is closer to CoTran's use of execution feedback as supervision than to full reinforcement learning, and it is cheap because each rollout is a simulator call.
Baselines
Naive reference. Tabero-VTLA with its English encoder swapped for an off-the-shelf multilingual encoder, given target-language instructions zero-shot with no target data. This measures how much force-word meaning comes for free from multilingual pretraining.
Strongest English reference point. Tabero-VTLA on the original English adverbs. The paper reports 87% success for firm grasps, 79% for gentle grasps, and over 70% average grip-force reduction (Tabero). We first reproduce these numbers in our harness. Any cross-lingual system is scored against our reproduction, not the paper's figure, to avoid harness drift of the kind BARS documents. No reported result exists on Lacuna for any non-English force-instruction policy, so there is no prior cross-lingual number to beat.
Translate-then-execute. Each target instruction is translated to English with a strong general multilingual LLM, then the English Tabero policy executes it. A second variant translates through the donor language first. This is the main competitor because it needs no robot data in the target language.
Execution-aware ablation. Our method with the second, execution-relabeling stage removed. Alignment then uses setpoint matching alone, which isolates whether execution feedback matters beyond setpoint agreement.
Donor-relatedness ablation. The same method with English as the donor instead of the related donor language, with matched pair counts.
Fairness controls. Every system gets the same tasks, 50 seeds per instruction, the same tactile readout, and the same adapter size and tuning budget. Translation baselines get a fixed prompt chosen on a held-out donor-language split.
What counts as beating a baseline. Our method must exceed translate-then-execute on ordering fidelity and on safe-gentle success in the low-resource targets. The improvement must have a confidence interval excluding zero under seed bootstrapping and must hold under both force readouts. Matching on firm-tier success is acceptable.
Why not just…?
Why not just translate to English and run the English policy? This is a real threat. For Hindi or Indonesian it may well match everything else. Our hypothesis concerns intensity gradations that English adverbs cover coarsely, such as reduplicated or particle-marked softening in Javanese or Bhojpuri. Here we expect translation to merge adjacent tiers. That would show up as ties in ordering fidelity even when each translation reads acceptably. The prior evidence is indirect: Can you map it to English? links multilingual failures to weak alignment with English concepts, and the Tatoeba Translation Challenge documents weak low-resource translation. The benchmark settles the question directly. If translation matches our method on ordering fidelity in the low-resource targets, we report that the translation pipeline suffices. The contribution then reduces to the execution benchmark and the demonstration that text metrics misrank systems, which is still worth having.
Why not just rely on a large multilingual encoder zero-shot? This is the naive baseline. Its failure is expected but not assumed: WikiBERT describes low-resource languages being starved of capacity in large shared models, and the zero-shot run measures how large that loss is for force words.
Why not just check that the policy outputs the right force setpoint, offline? Because the setpoint is not the outcome. Tabero's ultra-gentle runs had a plausible low force and still dropped objects. AnyTouch 2 found that models strong on static tactile classification failed on insertion. The execution-aware ablation measures how often setpoint agreement and execution agreement disagree. If they almost never disagree, the execution stage is unnecessary, and we will say so.
Decisive Checks
The first check tests the evaluation claim. We compute, across all systems and languages, the rank correlation between text-side scores (back-translation similarity, embedding cosine to the English instruction) and execution-equivalence rate. A strong correlation would mean execution adds little to text metrics, which would undercut the first claim. A weak or inverted correlation, especially within the gentle tiers, supports it.
The second check is the donor-relatedness ablation on ordering fidelity in Bhojpuri, Maithili, Javanese, and Sundanese. The second claim predicts that related donors beat English at equal data. Equal or worse results from the related donor would falsify it.
The third check is sensor robustness. The ranking of systems must be the same under the simulator's true force and under the tactile-estimated force, on both marker and marker-less sensors. A ranking that flips with the sensor turns the finding into a statement about sensor calibration, which TacEva-style reporting would then have to resolve.
The main risks are practical:
- Reproducing Tabero-VTLA from its description. If the reproduction falls well short of the reported 87/79 split, all comparisons are internal to our harness and labeled as such.
- The cost of collecting the human force-elicitation data, which we limit to about 20 speakers per language and 25 phrases.
References
- Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language
- XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval
- Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages
- Multilingual Argument Mining: Datasets and Analysis
- Cross-Tactile Sensor Representation Learning
- CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer
- Unsupervised Translation of Programming Languages
- CoTran: An LLM-Based Code Translator Using Reinforcement Learning with Feedback from Compiler and Symbolic Execution
- TacEva: A Performance Evaluation Framework for Vision‐Based Tactile Sensors
- BARS: Towards Open Benchmarking for Recommender Systems
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- Can you map it to {E}nglish? The Role of Cross-Lingual Alignment in the Multilingual Performance of {LLM}s
- The Tatoeba Translation Challenge - Realistic Data Sets for Low Resource and Multilingual MT
- WikiBERT models: deep transfer learning for many languages
- AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception