Lab of the Future: a robot arm for an autonomous lab

The brief. Autonomous robotics in the lab: a sample run with no person at the bench. A rack of tubes travels through a station lifecycle (drawer, centrifuge, pipetting, a storage hotel, parking) on an AMR. At each station a UR5e with a Robotiq 2F-85 gripper and a wrist RealSense does what a lab technician’s hands would: open the drawer, lift the centrifuge lid, align the rotor, load the rack, turn the knob, press the button.

The hard part. An orchestrator can only say “run the drawer station”. Turning that into a drawer that actually opens, every time, on instruments that move a few centimetres whenever someone docks the AMR differently, is the arm’s job. That was mine.

My role

I built the whole arm side:

• Perception: detection and 6D pose estimation of each station from the wrist camera, fused from the RealSense’s two IR images.
• Calibration tools: station, affordance and hand-eye calibration, so a moved instrument is a re-calibration, not a code change.
• The manipulation action composer: the engine every station skill runs on.
• The station skills: drawer, centrifuge lid, knob, rotor alignment, push button, rack pick-and-place.
• The robot control layer: UR5e motion over RTDE, my own socket client for the Robotiq gripper, and protective-stop handling.

Teammates built the orchestrator, the UI and the digital twin.

The manipulation action composer

The first version worked, but it was a demo: one robot, one camera, one set of stations, wired together. I pulled the generic machinery out into a standalone, installable package, so the next lab would be a configuration and not a fork.

What that buys:

• Stations are YAML. A pipeline is a list of steps with their parameters. Poses and file paths in the YAML are resolved automatically. Station pipelines run 4 to 10 steps; the orchestrator only ever needs the station’s name.
• Failures are visible. The composer stops at the first failing step, records which step and which action failed, and reports success: false to the orchestrator. A protective stop on the robot is an error, not something to carry on through.
• Swappable hardware. The robot, the camera, the vision algorithm and the messaging (MQTT or ROS 2) all sit behind small interfaces. The robot interface ships with UR5e, Franka and simulated implementations.
• Development without the cell. Three modes from the same code: a fully simulated robot and camera, the real UR controller software in a container, and the real cell. 21 tests run with no hardware at all.

One skill, step by step

Every station skill has the same shape, so a drawer, a lid and a knob differ only in their calibrated poses and parameters.

1. 1
   View

   Move the wrist camera to the station’s calibrated viewing pose.

2. 2
   Detect

   Detect the station and estimate its 6D pose from both IR images; keep the most confident detection and reject any above 2 px reprojection error.

3. 3
   Affordance

   Chain the station pose to the affordance, the exact thing the gripper touches: a handle, a knob, a button, a tube.

4. 4
   Act

   Approach along the tool axis, set the gripper width, follow the calibrated waypoints (a drawer pull, a lid arc), release.

5. 5
   Report

   Retreat, publish the messages the station hardware waits for, and report the result.

Some skills need more than a fixed motion. To load the centrifuge, the rotor has to sit with the right tube holes between its centre and the robot. The alignment action searches the rotor angle in 1° steps, grasps the centre handle, turns it and checks again.

Precision without a world model

There is no scene model of the lab. Precision comes from a short, calibrated chain instead: robot base → wrist → camera → station → affordance. Every link is measured, and each has its own tool:

• Stereo pose with a confidence. The wrist camera’s two IR images each give a pose; I triangulate the detection and score how well the two views agree on translation, rotation and ray geometry. A pose the two views disagree on gets a low score, so the pipeline stops cleanly instead of reaching for the wrong place.
• Station calibration. One root frame per instrument, estimated from several views. For the centrifuge, the rotor centre is fitted with a robust least-squares solve.
• Affordance annotation. A live tool where I move a coordinate frame with the keyboard until it sits on the handle or button, then save it. It replaced measuring grasp poses with a ruler. The rack’s eight tube poses are generated from one annotated centre.
• Hand-eye calibration. The arm scans a spiral around a ChArUco board, keeping clear of the UR5e’s wrist singularities. The tool solves with all five OpenCV methods, scores each on held-out views and prints the winner as a ready-to-paste config entry.

Moving a station means re-running its calibration, not editing code.

Where it is now

The arm ran the full sample lifecycle in the demo lab; one recorded full run lasted about 19 minutes. The composer has since been stood up on a second UR5e at another site, on a Jetson Orin as the robot’s computer, with the drawer station running there from the same package, new configs and no fork.