Rain on the windshield: a simulator, one network and the wiper

Camera-blindness detection for rain, end to end: 22,000 synthetic images with pixel-accurate rain masks, a classifier turned into a segmenter, and a sequence rule that survives the wiper.

A camera behind a wet windshield sees less, and the car should know when to trust it less. Camera blindness (blur, blockage, low sun, rain) was its own work stream. In April 2019 I wrote the first windshield-reflection and wet-road augmentations, which colleagues later grew into a larger blindness library. From October 2019 to January 2020 I owned rain, end to end.

No labels, so make them

Real rain doesn’t come with masks. So I built a rain and fog simulator:

Fog: Beer–Lambert attenuation, a guided filter, and the atmospheric light estimated from the dark channel.
Rain: streaks, heavy drops and droplets on the window, each with a pixel-accurate mask.

I rendered it onto clean drives from the US, Germany and China: 20,000 training and 2,000 test images, with 45% left clean.

One simulator gives both labels: the mask for segmentation, and “any rain pixel” for the image class.

From classifier to segmenter

The first version was a DenseNet-121 classifier trained from scratch at 960×1280 on 6 GPUs. A classifier can say “something is off” for the wrong reasons, so the second version became a segmentation network: an encoder-decoder with dense upsampling, plus the class head on the same features. The mask loss (weight 0.25) makes it learn where the drops are, not just that the image looks different.

One dataset trains both heads: the image label is simply whether the mask has any rain pixel.

The wiper problem

Frame scores oscillate with the wiper. Each wipe clears the glass for a moment, and the contrast change right after it hides the drops. A mean over the sequence gets pulled down by exactly the frames that matter least.

So the verdict is per sequence, built on peaks:

1
Drop the noise
Scores below 0.05 become zero.
2
Adaptive threshold
The threshold is the mean of the remaining non-zero scores.
3
Find peaks
Look for peaks above that threshold.
4
Decide
Rain if there is at least one peak per 50 frames.

One noisy frame can’t trigger it, and wiper dips can’t cancel real rain. The output is one JSON verdict per sequence, in the format the customer’s tooling expected.

Decide on sequences when the sensor has a cycle.

What I’d do the same way again: let the simulator produce the labels. The masks gave me segmentation and classification targets from one render.