AR-guided OEM parts inspection

AR glasses guide a worker through an engine checklist, a VLM judges each point, and my RGB-only tracker keeps the engine pose locked at 30 fps. Ten weeks, most of the system.

Goal. On an OEM line, a worker checks an engine against a checklist: parts, connections, wires, hoses, each ok or not ok. This proof of technology puts that checklist on Viture AR glasses: the glasses guide the worker to each point, a vision-language model judges it from a close-up, and the verdict is drawn back onto the glasses.

The core problem. Every overlay and every close-up depends on knowing where the engine is relative to the glasses, every frame, from a single RGB camera.

The inspection dashboard during a run: tracked pose on the live camera (top left), privacy-masked feed (below), checklist with verdicts, and the last close-up sent to the VLM. People in the camera view are blacked out for this post.

My role

Ten weeks, about 200 of the project’s 276 commits. I built:

The pose estimator: RGB-only 6-DoF tracking of the engine at 30 fps.
LightSpeedGlue: an FP16 TensorRT feature matcher that made template relocalisation real-time.
The platform: configs, ZeroMQ messaging hub, swappable cameras, record and replay.
The glasses app: an Android streamer for camera and head pose.
The middleware to the game engine that draws the overlay, the dashboard and the privacy mask.

Teammates built the VLM reasoner and the gate that decides when a view is good enough to judge.

System

One hub, every module connects out to it. Glasses, a phone, a depth camera or a recording can be the camera; the rest doesn’t notice.

Pose estimation

The constraints: no depth sensor, a big, nearly symmetric, metallic engine, and a wearer who walks right up to it for close-ups, so most of the engine leaves the frame exactly when the check happens. RGB-only deep pose estimators had already proved unreliable on it.

1
Register
Match the live frame against renders of the CAD model from 144 viewpoints and solve PnP. A weak match must be confirmed by a second, independent template, so the engine’s near-symmetric views can’t lock onto the wrong side.
2
Track
Features tracked with KLT and lifted onto the CAD model as 3D landmarks, seeded by the head’s gyro, solved with RANSAC PnP and smoothed with a One-Euro filter. About 17 ms per frame on a DGX Spark.
3
Correct
Every 5 frames an async worker re-solves against the templates and corrects the drift, without blocking the 30 Hz loop.
4
Close-ups
When too little of the engine is in view to verify a pose, a SIFT frame-to-frame fallback carries it.
5
Latency
The estimator measures its own processing time and predicts the pose forward by that much from the gyro, so the overlay doesn’t trail the engine.

Realistic templates mattered as much as the algorithm. Matte renders answered on 42–70% of frames; a realistic bank, rendered in Blender and aligned to the CAD frame, answered on 87–98%.

Making templates real-time

PyTorch LightGlue took about 1.4 s per cold start, far too slow to correct a 30 fps tracker. LightSpeedGlue splits the work: encode the live frame once (about 33 ms), then run a matcher-only FP16 TensorRT engine per template (about 7 ms).

3.45× faster per template than PyTorch.
About 29 ms per solve with the previous pose as a prior.
A cold sweep took about 78 ms on the DGX Spark, against about 777 ms on a Jetson AGX Orin.

Validation

For ground truth I put AprilTags on the table and recorded nine test sequences. The tags were never an input to the estimator; they only scored it. Because the record and replay layer plays a recording back exactly, every change was measured on the same nine sets before it went near the live system.

The tags are coplanar, and a planar target has two near-identical pose solutions when you face it head-on: on one recording the reference swung by 12–31°. I solved the reference from several starting points, kept the answer consistent with the previous pose, and dropped frontal frames whose rotation uncertainty was over 3°. After that, no jumps over 12° on any set.

Pose estimator, 9 recordings	Rotation	Translation
Median error per recording	1.6–3.7°	22–43 mm
Close-up fallback, hardest recording	3.75°	37 mm

Streaming from the Viture glasses

Me wearing the Viture glasses and neckband, with the engine pose tracked on the glasses' camera stream on the laptop.

Head pose. The Viture neckband’s own service already owned the USB connection, so every SDK route crashed or stole the device. The vendor’s service interface for Unity streams head orientation at about 100 Hz without touching USB.
Video. Hardware H.264 instead of JPEG: about 10× smaller frames, with every IMU sample matched to its frame on one clock. On the host, one persistent decoder took decoding from about 1 fps to the full 30.
Calibration. Camera intrinsics at 0.42 px reprojection error; camera-to-IMU rotation measured from optical flow.

The same app later ran on a phone.

Privacy

The privacy mask blacks out everything outside the engine and removes the wearer’s hands and arms with a hand-landmark model, about 16 ms per frame on the GPU. If the pose is lost, the whole frame goes black. A hand model beat a full-body one: the body model had no GPU path and kept detecting people in the background.

Results

RGB-only engine tracking at the camera’s 30 fps, 1.6–3.7° and 22–43 mm median error.
Template relocalisation fast enough to correct the tracker every 5 frames.
A system where glasses, a phone or a recording are interchangeable, and every change is benchmarked on replayed data.