Six years of perception for cars

ML engineer, then head of perception at ESR Labs, 2017–2022. Small, fast networks for the embedded hardware in cars, the path that got them there, and one study that didn't work.

From late 2017 to 2022 I built perception for cars at ESR Labs: networks that see lanes, people, rain and depth, and that run in real time on embedded hardware in the car. I joined as an ML engineer and became head of perception in 2020. Most of the code is internal, so each piece of work has its own post. This one is the map, and what connects them.

The role

As head of perception my job was what the car should see, and getting the networks that see it onto the embedded hardware in the car. In one picture:

Schematic. Each sense in this scene is a network I built; each has its own post below.

The work, in order

2017–18
Real-time segmentation
A config-driven training framework, my own fast segmentation network, and a one-step export to TensorRT. Read →
2018
Detection in video
Anchors sized for pedestrians and features borrowed from neighbouring frames: 0.796 mAP on 81,845 test frames. Read →
2018
Camera localisation
Curbs from the camera matched against an HD map on Drive PX, and the debugger that showed a tilted corner was fooling it. Read →
2019
Rain detection
A rain simulator with pixel-accurate masks, and a rule over the image sequence that survives the wiper. Read →
2021
Depth completion
Sparse LiDAR plus camera to dense, metric depth, in the car’s demo. Read →
2021
Panoptic segmentation
Every pixel gets a class and every object its own mask, rebuilt in PyTorch and slimmed for the car. Read →
2022
Perception trilogy: one encoder, three tasks
Depth, segmentation and instances from one network, with learned loss weights so no task drowns the others. Read →

What leading meant in practice

Make networks cheap to try. In 2017 that was the config framework: a new network is a plugin class and a JSON file. In 2020 it was our PyTorch perception repo. Segmentation was its first project, and its base model, config loader and training loop became the template for everything after it. I made 244 of its 519 commits.

One road to the car. Every network took the same path: PyTorch, then ONNX or TorchScript, then a TensorRT engine in FP16 or INT8, then one C++ engine that runs it behind the same interface as everything else. I added demo deployments with default configs, so anyone on the team could run a network on the car’s hardware without asking me.

Ship it as a demo. In January 2022 the car ran a demo across its two Drive AGX units: segmentation, stereo depth and object tracking, plus a traffic-sign detector that uploads images to the cloud when its detections look poor. I packaged my three networks for it (segmentation, mono depth and depth completion) with their configs and handed them to the colleague who ran it.

Bridge the customer’s code and ours. In 2018, before I led the team, the detection framework came from the customer. I imported it, kept both code bases in sync, and fixed what blocked the team: dropped batch-norm statistics, mismatched augmentations, a wrong multi-GPU metric.

Habits that outlived the models

Read your labels as the right type. Cityscapes disparity PNGs are 16-bit; read as 8-bit, the ground truth was silently corrupted. And a horizontal flip is not a free augmentation for traffic signs: it changes what a direction sign means.
Fix the metric before you trust it. The detection KPI applied NMS before scoring and hid what the detector really did.
Profile before you optimise. A trace of the deployed graph found the slow ops in panoptic segmentation; the mask head went from 256 filters to 32.
Calibrate for the precision you deploy. On Fast-SCNN, plain INT8 cost 5 mIoU points (0.583 → 0.533). With calibration on 100 images, INT8 matched FP32 exactly.
Build the debug tool. A week on a localisation debugger turned “the score is wrong” into a picture of what the camera saw and where it landed on the map.
Small wins compound. In 2017 a capsule network for traffic signs trained about 10× faster once I fixed its routing for all but the last iteration and ran three iterations instead of four.

A study that didn’t work

In 2019 I tested whether a neural network could read a car’s ultrasonic parking sensors better than the classic geometry. Four sensors on a bar at bumper height, each sending a ping and listening for echoes, in a test room with poles and cardboard boxes.

An echo only tells you how far away something is, not in which direction. So each echo puts the obstacle somewhere on an arc around the sensor, and where the arcs from several sensors cross, that’s the obstacle.

Schematic. Echoes give distance, so position comes from where the arcs cross. The echo carries little about the object’s shape.

I tried five networks in stages, each changing one thing: a plain dense network, 1-D convolutions over the raw echo signals, the same plus the drawn arcs as a second input, the drawn arcs alone, and a version that only had to find obstacle centres.

The networks could trace where an obstacle was. None of them could tell what kind of obstacle it was from the echoes alone. At four sensors, an echo says a lot about distance and very little about shape. We stopped there, wrote down why, and moved the effort to where it would pay off.

A clear “no” with the reason is a result too.

The thread

Small, fast networks, measured on the hardware they will run on, with tools that make failures visible. It is the same thread in my robotics work now.