A voice agent for a cobot: LLM tool calling on a real arm

Speech in, one forced tool call out, and a robot-agnostic skill layer that does the rest. How I made an LLM agent useful on a real robot without letting it improvise.

I built a speech-driven agent for a cobot: you talk to the robot, the agent works out what you want, and the arm finds the object, estimates its pose and picks it. The agent can only ask for what the robot can actually do; everything physical sits behind skills I wrote.

The Franka FR3 cell with the test objects.

The agent loop

The agent loop. The model never sees a motor; it fills in one structured call that my code checks before anything moves.

“Can you pick up the camera holder?” → Whisper → the LLM calls dispatch_action → the call is checked → the arm picks → “I picked up the camera holder.”

One tool, and it is forced

The LLM gets exactly one tool, and tool_choice forces it to use it. No free text, no plans to parse.

dispatch_action(language, action, object, feedback)
  language   the language the user spoke
  action     one of a closed list: pick, place, drop, open
  object     one of the objects in the database
  feedback   what the robot says back to the user

The allowed object names are injected into the system prompt from the object database, so the model knows what exists in this cell today. The prompt keeps it to scanning, picking and placing, at temperature 0.2. Indirect requests still work: say you are coughing, and it picks the cough drops, because they are in the database.

An indirect request: I say I'm coughing. Cough drops are in the database, so it picks them up.

Safe by default

The agent is the boundary between a probabilistic model and a physical robot, so every path that isn’t clearly valid ends in doing nothing:

Safe parsing. An answer that doesn’t parse becomes none. An action outside the list becomes none. An object that isn’t in the database becomes none. The model cannot send the arm after something that doesn’t exist.
A fallback for weak tool calling. On local models, if the tool call fails, the agent re-asks in JSON-only mode with the same contract.
Sanity checks in the skills. A pose estimate further than 0.8 m or closer than 0.15 m from the camera is rejected before any motion.
Recoverable failures. If the best grasp is unreachable, the skill tries the next annotated grasp. After a robot exception it re-initialises the robot instead of crashing.

It answers in your language

The tool call carries the language the user spoke. The robot’s result comes back in English, a second prompt translates it, and TTS speaks it. No per-language code.

Asking for the remote in Telugu, my mother tongue. It understood, but GPT replied in Hindi. Motion is slowed for testing.

The skills behind the tool

The tool is only as good as what it calls. Three parts do the physical work:

A robot-agnostic control layer. Pick, place, drop and scan, with motion, kinematics and execution inside. The robot class is loaded by name from config, so the same agent runs on a Franka FR3, a Trossen Viper 300s and a UR5e, and on a dummy robot at a desk.
Perception from a mesh. CNOS finds the object from templates rendered from its CAD model; FoundationPose gives its 6D pose. Onboarding a new object needs a mesh, rendered templates and a database entry, no training. The database is re-read on every detection, so a new object shows up without a restart.
Grip annotation. A tool where I place grasps once on the object’s mesh. At runtime they are moved into the scene with the estimated pose, scored by how close they are to vertical, and tried best first.

The grip annotation tool, with several grasps on one object.

From cloud to local

The first version ran on GPT-4 and OpenAI’s speech APIs, later on GPT-4o mini through Azure OpenAI. In 2026 I moved the whole agent off the cloud onto local OpenAI-compatible servers on a DGX Spark: Whisper large-v3-turbo for speech, Qwen3 8B on Ollama as the LLM, and a local TTS server.

The contract didn’t change: same tool, same validation, same skills. Switching is one import. The demo now runs with no cloud account and no internet, which matters when the objects on the table are someone else’s.

Stress test

To see how the skills hold up over hours, I ran a pick-and-drop loop without the language layer: pick an object, drop it with a random yaw, find it again, repeat until 10 failures in a row.

Run 1: 133 of 153 picks (86.9%) in about 1 h 15 min, until the gripper fingers froze half-open.
Run 2: 302 of 342 picks (88.3%) in about 3 h 8 min.

Most failures were gripper errors, plus a few slips from small pose errors. Perception was rarely the weak link.

A teaser

I showed this demo at NVIDIA GTC Paris, and Jensen Huang stopped at the booth. That story is in the next post.

References: CNOS · FoundationPose · OpenAI function calling · Ollama