Depth completion: mono depth plus a free depth channel

I gave our real-time mono depth network one extra input channel for sparse depth from LiDAR, stereo or whatever the car has. Same cost as mono, metric output, and it ran in the car.

On the embedded hardware in a car, dense depth had two options, and neither was good enough.

Stereo networks build a cost volume: match features over every candidate disparity, then run 3D convolutions over it. Accurate, but expensive, and not real-time on the hardware in the car.
Monocular networks are cheap, but their scale is ambiguous and they guess depth from appearance alone.

The car already measures some depth: LiDAR points, or disparity from classic stereo matching. It’s sparse, but it’s metric. So my idea was simple: keep the mono depth network and add that sparse depth as a fourth input channel.

And its sparse LiDAR scan lines, the fourth input channel.

Output: dense depth for the whole frame (brighter is closer).

Almost free

The network is the one I already ran for mono depth: a ResNet-18 encoder with a light decoder at 192×640. The only change is the first convolution, which now takes RGB plus sparse depth (4 channels instead of 3). Everything after it costs exactly what mono depth costs.

What the extra channel buys:

Metric depth. Evaluation uses no median scaling. The depth has to be right in metres on its own.
Any depth source. The network just sees a sparse depth map. I trained with LiDAR; the deployed config takes a sparse depth map synced to the image within 0.1 s, and it also had an input for stereo-matched disparity.
A fallback for free. With the depth channel empty, the same network is a mono depth estimator.

Compared with stereo networks with cost volumes, it held its own on quality and was the one that actually ran in real time on the embedded target. The depth-completion model was part of the in-car demo on the Drive AGX, next to segmentation and tracking.

Where it came from

I started with self-supervised mono depth in 2020, on A2D2 and KITTI, moving from PackNet to the lighter ResNet-18. That mono checkpoint initialised the depth-completion network, so the step from mono to completion really was one layer.

How it learns without labels

Dense depth labels for driving data barely exist, so the dense supervised loss is switched off.

1
Predict depth
ResNet-18 encoder and light decoder, 4-channel RGB + sparse depth input at 192×640.
2
Predict motion
A PoseNet estimates the camera motion between frames.
3
Reproject
Warp neighbouring frames into the current one with depth and pose; the photometric error is the loss.
4
Anchor the scale
A velocity loss ties PoseNet’s translation to the car’s measured speed, so the depth comes out in metres.

Pose limits everything

View synthesis is only as good as the camera motion behind it.

Integrating vehicle bus signals drifted.
A vehicle-to-camera axis convention bug quietly broke the warps until I found it.
Trajectories from visual SLAM gave the best configuration.

Self-supervised depth is a pose problem in disguise.

The scene isn’t static

This is the fundamental problem of self-supervised depth in traffic. The photometric loss assumes a static world: when the camera moves, every pixel should shift by an amount set by its depth. The car directly ahead breaks that. It drives at about our speed, so its pixels sit in exactly the same place from frame to frame. The only static-world explanation for a pixel that never moves is a point infinitely far away, so the network learns a hole where the car is.

The car ahead moves at our speed, so its pixels don't move between frames. In the depth map (brighter is closer) it comes out as a dark hole, as if it were far away.

Same effect in town: the car ahead in our lane turns dark, while the parked cars on the right get sensible depth.

Parked cars are fine; they are static, just like the road. It is only the vehicles moving with us that the loss can’t explain. So I took the pixels the segmentation marks as dynamic objects out of the photometric loss, but only in the central band of the image (37.5–62.5% of the width), where the ego-lane car lives.

Don’t let it lean on the sensor

If the depth input is always there, the network learns to copy it and stops looking at the image. So training randomly drops points, and in 50% of samples drops all of them. Ground truth is saved before dropping, so validation measures how well it fills in points it never saw.

Masking movers, then adding tasks

The last experiments ran from December 2021 to February 2022.

Velocity weight. I swept the weight of the velocity loss. Only 0.004 trained stably (best abs_rel 0.150–0.155). At 0.01, 0.006, 0.0025 and 0.001 the runs hit their best early and then degraded to 0.21–0.59.
Moving-object mask. Taking dynamic-object pixels in the central band out of the photometric loss brought the best abs_rel from 0.1498 to 0.1354 (a1 0.862), about 10% better.
Three tasks in one network. Starting from the depth-completion checkpoint, I fine-tuned one ResNet-18 network for depth, semantic segmentation and instances together. Depth improved slightly to 0.1326 abs_rel (a1 0.864), with about 0.42 mIoU and 0.31 mAP@50 on the other two tasks.

Each bar is the best epoch of its run, December 2021 to February 2022.

What I’d watch for

I found and fixed a validation leak and a shuffling bug along the way. Both made numbers look better than they were. If a depth result looks too good, check the split before the model.