Detecting cyclists in video, then tracking them

In 2018 the task was vulnerable road users: pedestrians, cyclists and motorcyclists in grayscale front-camera video, for a German OEM’s driver-assistance programme. Frames were 960×1280; the training set had about 170,000 images.

The detection framework came from the customer’s side. I imported it into our repo, kept the two code bases in sync, and fixed what blocked training:

• batch-norm statistics were silently dropped when warm-starting from a checkpoint;
• image and label augmentations drew different random values, so they could disagree;
• MeanIoU was wrong under multi-GPU training because it was averaged across GPUs;
• the SSD loss broke on newer TensorFlow; fixing it let the team move to 1.6.

SSD made for people

I ported SSD into the plugin structure, so any encoder shares one loss and one set of summaries. Its default anchor ratios were 2, 0.5, 3 and 5: mostly wide boxes. Pedestrians are tall, so I changed them to 2, 0.5, 1/3 and 1/5. I also removed the square-input assumption, so full 960×1280 frames no longer had to be squashed.

Borrow from the neighbours

A single frame can be blurred, occluded or badly lit, while the frames around it see the object clearly. In February and March 2018 I implemented flow-guided feature aggregation (Zhu et al., 2017) end to end in TensorFlow:

• All frames go through the same encoder as one stacked batch, so the weights are shared.
• A FlowNet-style network estimates the flow, rescaled to every feature-map resolution before warping.
• Warped features are embedded and compared with frame t by cosine similarity; a softmax over the frames turns that into per-pixel weights.

Training used one reference frame within ±3 frames; inference used all seven. That works because the softmax normalises over however many frames arrive.

Once it worked on SSD I factored it into a detector-agnostic module. It went onto Faster R-CNN in May 2018, and a year later onto the camera-blindness models.

The cost is memory. Every frame is another pass through the encoder, and in the Faster R-CNN version the flow network alone was 37.1 of 44.5 M parameters (83%).

Fix the metric, then trust it

The KPI script applied NMS before scoring, which hid what the detector actually output. I made NMS off by default in April 2018; the numbers below come after that fix.

The best model was a Faster R-CNN with a DenseNet backbone and the flow module, fine-tuned to epoch 40. On 81,845 test frames it reached 0.796 mAP, with an object-level detection rate of 0.956 (frame-level 0.766).

A NASNet version with flow (147.2 M parameters) was still at epoch 17 when the comparison was made, so it says nothing either way.

Segmentation as a second opinion

Segmentation networks already marked “pedestrian”, “bike” and “car” pixels. I tried two ways to use that:

• Learned fusion. Concatenate the segmentation map into every SSD feature scale. My first version added it at the input; moving it to every scale left the pretrained encoder’s input untouched.
• A rule. Drop a box if less than 25% of its pixels belong to those classes in a separate segmentation network’s output. It is cheap and works with any detector.

The repo doesn’t keep the KPI comparison for these two.

Tracking (2019)

In 2019 I wrote the tracker, in the style of Deep SORT:

• an 8-D constant-velocity Kalman filter per box, with noise proportional to box height, because a taller box is closer and moves more in pixels;
• a matching cascade by track age and Hungarian assignment, gated by Mahalanobis distance and appearance similarity, with an IoU fallback;
• a gallery of 100 appearance descriptors per track;
• three swappable appearance features: a histogram that works on grayscale, a re-id CNN, and MatchNet.

It came with unit tests and MOT metrics. Two bugs worth passing on: the munkres solver modifies its input matrix, and popping from a list while iterating over it skips elements.

What I took from it: fix the evaluation before trusting it, and budget memory before adding frames.