Who decides where to look: convolution vs attention

In a convolution, the routing is done by the architecture. In a transformer, it is done by self-attention. One idea that explains locality, data hunger, the n² cost, and why you can't cache a matcher.

Every layer of a network does two jobs. It decides where to read from: which inputs feed this output. And it decides what to compute from what it read. I call the first job routing.

The difference between a convolution and a transformer is mostly about who does the routing. In a convolution, the routing is done by the architecture. In a transformer, the routing is done by self-attention, from the content itself.

Toy scene, one query cell moving around. Left: the convolution always reads the same 3×3 window. Right: attention reads from the cells that look like the query, near or far: the other wheel, the other lane marks.

Two formulas, one difference

A convolution computes output i from its neighbourhood:

y_i = Σ_{j ∈ N(i)} W_j−i · x_j

The weight depends only on the offset j − i. Which inputs feed output i was decided when someone chose the kernel size, the stride, the dilation and the depth. It’s the same for a photo of a cat and a photo of a car. The weights learn what to compute; they never change where to look.

Self-attention computes output i from every input:

y_i = Σ_j a_ij · V x_j, with a_ij = softmax_j(q_i · k_j / √d)

and the query q and key k are both computed from the input. So the weight between i and j is a function of x_i and x_j. The routing is recomputed for every input.

The routing table

Write down, for one layer, how much each output reads from each input, and you get a matrix: the routing table.

Two different inputs of 16 tokens (coloured strips). The convolution's table is the same band for both. The attention table follows the content, and changes with it. Toy attention: same class scores high, distance costs a little.

For a convolution that table is a fixed band, set before training. For attention it is a fresh, dense matrix for every image. Most of what is good and bad about each follows from that.

Reaching far costs depth, or n²

Two wheels of the same car can be hundreds of pixels apart. A 3×3 convolution can’t connect them in one layer. It needs depth or pooling until the receptive fields meet. Attention connects them in one layer, if their content says they belong together.

A convolution reaches far by stacking layers. Attention reaches far in one layer, and spends most of its weight where the content matches.

The price is the table itself. A 640×640 image cut into 16-pixel patches is 1,600 tokens, so one attention head in one layer scores 2.56 million pairs. A 3×3 convolution reads 9 inputs per output, whatever the image size.

What the architecture gave for free

Fixing the routing is not a limitation only. It’s an assumption, and a good one for images: nearby pixels belong together, and a pattern means the same thing wherever it appears. A convolution gets both for free. Because the same weights slide over every position, a shifted input gives a shifted output.

Attention starts with neither. Without positional encodings it doesn’t even know which token is where: it treats the image as a set of patches. It has to learn locality from data. That is why the original Vision Transformer fell behind comparable ResNets when trained on ImageNet’s 1.3 million images alone, and only overtook them when pre-trained on 14 to 300 million.

It also works the other way round. A multi-head self-attention layer with relative positional encodings can express any convolution (Cordonnier et al., 2020). A convolution is attention whose routing has been frozen to relative positions.

Convolution decides where to look before it sees the image. Attention decides after.

The space in between

It isn’t a binary choice. Most of the interesting architectures of the last years move the routing a step to the left or right.

Deformable convolutions let the content bend a fixed window. Windowed attention puts an architectural fence around content-based routing. Both trade some flexibility for cost.

The idea is older than transformers. Capsule networks (2017) routed between layers by agreement: a lower capsule sends its output to the upper capsule that agrees with it, found by a few iterations per input. I trained one on traffic signs that year. The routing iterations were the expensive part, and fixing the coefficients for all but the last iteration, with three iterations instead of four, made training about 10× faster. Content-based routing has always cost compute.

Rule of thumb

If you know the structure, like locality or a symmetry, put it in the architecture. It’s free at run time and it saves data.
If you don’t know which parts relate, let attention compute it, and pay in data and in n².
When you deploy, remember where the routing is. Anything that depends only on one input can be cached; anything that routes between inputs can’t.

References: Attention is all you need · ViT · Self-attention and convolution · Deformable ConvNets · Swin · Capsules