Words and Forces as Bids for Initiative in Robot Co-Manipulation

When a person and a robot carry, align, or fasten something together, they constantly trade who is in charge. Sometimes they trade by speaking ("let me guide it," "hold it right there"). Sometimes they trade by pushing: a hand leans on the part, or resists the arm. Today's systems handle these two channels separately. Dialogue systems for collaborative robots negotiate who does which step, then execute it with stiff position control. Compliant controllers respond to force but never hear what the person said. We propose one shared state for both channels: a running record of who currently holds initiative along each direction of motion. Speech and force both update it, and it sets the robot's physical stiffness. An utterance can create a lasting physical commitment, such as "stay rigid vertically until the screw is in." A push can create a conversational obligation, such as acknowledging the push, yielding, or explaining why the robot will not yield.

The claim is narrow and testable. For simple one-word commands, a lookup table from keywords to stiffness settings will likely be enough. Coupling the two channels should matter in the cases where each channel alone is ambiguous. One case is a deictic or partial instruction ("a bit more that way") that only makes sense together with the force direction. Another is an axis-specific commitment ("keep it level but let me slide it"). A third is a channel conflict: the person says "careful" while pushing hard. We predict that a shared initiative state with explicit commitments reduces physical "fighting" between human and robot, and reduces repeated confirmation questions, on exactly these cases. We also predict it gives no gain on simple commands.

Background and the gap

Variable impedance control treats the robot's hand as a virtual spring and damper, $F = M\ddot{e} + D\dot{e} + Ke$. Here $e$ is the deviation from the planned position, $K$ is stiffness, $D$ is damping, and $M$ is virtual inertia. The problem is then choosing $K$ and $D$ over time. Several lines of work supply the method we reuse. VICES shows that commanding target poses and gains from a slow outer loop (20Hz), over a fast inner impedance loop (500Hz), is a good interface for learning. It also shows the interface transfers across robots. Learning Collaborative Impedance-Based Robot Behaviors learns stiffness matrices for a human-robot table assembly from demonstrations. It separates a compliant repositioning phase from stiff holding phases, with a stiffness ratio of 10.12. That work is purely haptic and reactive, and it motivates our question of what speech adds. CompliantVLA-adaptor is the closest system. A vision-language model infers the task phase at about 1Hz and outputs direction-dependent $K$ and $D$, while a 1000Hz loop cuts stiffness when force exceeds a hand-tuned 30N. It has no dialogue, and its safety cutoff is global rather than negotiated. Our design keeps its two-rate split but replaces "phase inferred from pixels" with "initiative inferred from speech and force."

On the dialogue side, MICoBot is the strongest evidence that structured mixed-initiative dialogue helps physical collaboration. It reports 77.8% task success against 27.8% for a pure LLM baseline. It also reports that explaining its physical limits raised human acceptance of help requests from 55% to 86%. Its dialogue decides only who does each step, in turn. The authors name concurrent hand-over-hand co-manipulation as outside its scope, and its controller has no compliance. We target exactly that excluded regime. Two older ideas supply our dialogue mechanism. Importance-Driven Turn-Bidding lets each party place a bid between 0 and 1 for the conversational floor, and it beat keep-or-release turn-taking (mean policy cost 5.52 vs. 6.01). Discourse Obligations in Dialogue Processing keeps social obligations, such as having to answer a question, on a stack above private goals. This produces mixed initiative without global replanning. We extend both to the physical channel: a force is a bid, and an unanswered push is an open obligation.

Inference-time alignment and multi-agent decomposition act here as constraints, not as the core. Pluralistic AI Alignment Requires Inference-Time Multi-Objective Control separates negotiable trade-offs from non-negotiable constraints, and asks that runtime steering stay inside a region where safety is guaranteed. We adopt that split literally. How assertive the robot is, and how quickly it yields, are negotiable in conversation. Force and stiffness envelopes are not negotiable and are enforced below the language model. The decomposition into separate roles follows from the timing problem that CompliantVLA-adaptor exposes. A language model at 1Hz cannot respond to a push in time, so haptic bids must be handled by a fast non-LLM component. The language model only handles speech and the commitments speech creates.

Core Claim

Let $h_i(t) \in [0,1]$ be the human's share of initiative along motion axis $i$, where 0 means the robot leads and 1 means the human leads. Stiffness is set as

$$K_i(t) = \Pi_{\mathcal{S}}\big(K_i^{\min} + (1 - h_i(t))(K_i^{\max} - K_i^{\min})\big),$$

where $K_i^{\min}$ and $K_i^{\max}$ are the task's stiffness range on that axis. $\Pi_{\mathcal{S}}$ is a projection onto a safety set $\mathcal{S}$ of allowed stiffness and force pairs, which conversation cannot change. Damping follows $K$ at a fixed damping ratio.

The value $h_i$ is updated at two rates. At 1kHz, sustained force that opposes the robot's motion along axis $i$ raises $h_i$, unless an active commitment pins that axis. At about 1Hz, a language agent turns each utterance into a commitment: an axis set, a target value of $h$, a release condition, and whether it overrides haptic bids. A physical bid that changes $h$ by more than a threshold pushes an obligation onto the dialogue stack. The robot must then acknowledge the push, yield, or explain its refusal. A physical bid that conflicts with an active commitment triggers a clarifying question instead of silently yielding or resisting.

What is new is this pair of rules: commitments bind speech to later physical behavior, and obligations bind force to later speech. Neither MICoBot nor CompliantVLA-adaptor has either rule.

Research Program

We would build four components with fixed interfaces. First, a haptic bid detector: a filtered estimate of the external wrench, compared against the expected wrench of the current motion. It runs at 1kHz and has no learned language component. Second, a commitment agent: an LLM whose output is restricted at decode time to a small grammar of commitments over axes, targets, and release conditions. The grammar is enforced by masking invalid tokens, which is the simplest form of inference-time control over logits. The permitted numeric ranges are clipped to $\mathcal{S}$ before they reach the controller. Third, an obligation manager that holds open obligations from both channels and decides what the robot says next. Fourth, the impedance scheduler above.

The negotiable preference is how readily the robot yields. It is set by one scalar gain on the haptic update. The user can change this gain in conversation ("don't fight me so much"). That turns the pluralistic-alignment split into something measurable: a change in preference should move yield latency while leaving peak force inside $\mathcal{S}$.

The analysis would stratify interactions into three kinds. Simple single-channel commands form the first. Compositional or axis-specific commands form the second. Cross-channel conflicts or ambiguities form the third, where the utterance only resolves given the force, or the two channels disagree. Most of the evidence comes from comparing results across these strata.

Baselines

The naive reference is fixed medium stiffness plus a 30N force cutoff with no dialogue. This is the reference point that Learning Compliant Stiffness by Impedance Control-Aware Task Segmentation and Variable Impedance Control - A Reinforcement Learning Approach argue is inadequate. We include it to size the problem.

The strongest existing method for setting impedance with a language model is CompliantVLA-adaptor. It reports average success rising from roughly 9.8% for rigid VLA baselines to 17.3% under a strict 30N constraint across eight LIBERO and ManiSkill contact-rich tasks. We run it two ways, as published and with the speech transcript appended to its prompt. The second version is the fair "just tell the VLM" competitor. It uses the same LLM backbone as our commitment agent, the same 1000Hz reflex, and the same prompt-tuning budget. Beating it means lower fighting power and fewer force violations on the compositional and cross-channel strata, at equal or better task success.

For dialogue, we reimplement a MICoBot-style sequential allocator over the same tasks, with position control and turn-based handover (MICoBot). Its reported 77.8% comes from different tasks on a TIAGo robot. No reported result was found on Lacuna for it on concurrent co-manipulation. We expect it to be competitive on success rate but slower, because it cannot share a part concurrently.

The haptic-only competitor is the task-parameterized stiffness model of Learning Collaborative Impedance-Based Robot Behaviors, trained on the same demonstrations we use. We also run a keyword rule table mapping phrases to fixed stiffness presets, combined with the same haptic detector. Two ablations isolate the key ingredient. One removes commitments, so $h$ is set only by the most recent event. The other removes obligations, so pushes are never verbally acknowledged. A third ablation replaces grammar-constrained decoding with free-text output that is parsed afterward. It tests whether the logit-level constraint matters for commitment validity.

Why not just…?

Why not just use a keyword-to-stiffness lookup table? For "hold" and "let go," it will probably match us, and we will report that honestly. A table cannot encode release conditions, per-axis targets, or the reading of a deictic phrase from the force direction. The stratified evaluation settles this. If the table plus the haptic detector matches us on the compositional and cross-channel strata as well, the commitment mechanism is unnecessary and the paper's claim fails.

Why not just give the transcript to a vision-language model that already sets impedance? This is the real threat. It is the transcript-augmented version of CompliantVLA-adaptor. Its weakness is structural. It re-derives stiffness from scratch at about 1Hz, with no memory of what the person asked the robot to maintain. It also has no path from force to speech. Pushes are handled only by the reflex that cuts stiffness above 30N, which cannot tell "the person wants to lead" apart from "something is jammed." We expect this baseline to violate commitments under disturbance and to leave pushes unacknowledged. We measure both directly.

Why not just detect intent from force alone and skip speech? Reactive haptic schemes like Learning Collaborative Impedance-Based Robot Behaviors work when the phase structure matches the demonstrations. They cannot receive an instruction whose effect lies in the future, such as holding rigid while the person fetches a screw. They also cannot tell an accidental bump from a deliberate bid. Its cross-channel failures will show up as either over-yielding during commitments or resisting deliberate guidance.

Why not just train an end-to-end policy? Rigid VLA policies collapse under force limits, reaching roughly 9.8% in CompliantVLA-adaptor. We found no speech-conditioned co-manipulation data large enough to train one. We therefore treat this as out of scope rather than as a baseline.

Proposed Experiments

The main testbed is a simulated co-assembly setting built on RAMP frames in Isaac Sim. A scripted human partner holds or guides frame members, applies scripted forces, and speaks from a template grammar with paraphrases produced by an LLM. The autonomous REBA baseline reaches 84% completion on Easy structures in 580 seconds, with peg insertion as the main failure mode. That anchors task difficulty, although it is not a collaborative number. As a controller sanity check, we reproduce the eight LIBERO and ManiSkill contact-rich tasks from CompliantVLA-adaptor without a partner. The goal is to confirm that our scheduler does not fall below the reported 17.3% under the 30N limit.

The metrics fall into four groups. The first is task success and completion time. The second is peak interaction force and the force-violation rate. The third is "fighting," defined as integrated negative mechanical power between the human's force and the robot's velocity. The fourth covers yield latency after a deliberate bid, commitment-violation rate under disturbances, unacknowledged-push rate, and the count of repeated confirmation questions. All of these are reported per stratum.

A real-robot study on a Franka Panda follows. It uses a two-person-style table-fastening task modeled on the KUKA setup of Learning Collaborative Impedance-Based Robot Behaviors, with participants and a within-subject comparison against the two strongest baselines.

Three decisive checks determine whether the idea holds. First, on the cross-channel stratum, the full system must reduce fighting power and commitment violations relative to the transcript-augmented CompliantVLA-adaptor, with no rise in force violations. Second, the no-commitment and no-obligation ablations must each lose most of that gain. If neither does, the gain comes from somewhere else. Third, a user's conversational change to yield readiness must shift yield latency, while peak force stays inside $\mathcal{S}$ in every trial. That result is the concrete test of keeping negotiable preferences separate from non-negotiable safety. If the rule table ties the full system on all strata, we would report that result and drop the commitment claim.