What the model is made of.

A neural network does not natively understand a kitchen photo or the sentence pick up the sponge. It understands vectors of numbers. The first job of the model is to turn both into the same kind of thing — a flat sequence of vectors we call tokens.

For text, a tokenizer chops a sentence into sub-word units (pick, up, the, spo, nge) and looks each one up in a fixed dictionary. Every entry has a learned vector, typically a few hundred numbers wide.

For images, the trick is grid-and-flatten. A 224×224 photograph is sliced into a 14×14 grid of 16-pixel patches; each patch runs through a small network and is squeezed into a single vector of the same width as a text token. A photo becomes 196 vectors. ¹ Both streams concatenate into one long row.

Attention: how a token decides what to listen to.

Once the sequence exists, tokens have to talk to each other. The mechanism is called attention, and it is mechanical.

For every token, the model produces three short vectors from its embedding: a query, a key, and a value. The query is a question (“what kind of word goes after cat?”); the key is a label the token advertises (“I am a verb”); the value is content the token will hand over (“sat”).

To update a token, the model dot-products its query against every other token's key, normalises the scores with softmax so they sum to one, and uses those weights to take a weighted average of the values. That weighted average is the token's new representation.

The $\sqrt{d}$ keeps dot products from blowing up when vectors get wide; without it, softmax saturates and the model stops learning.

Multi-head attention: several questions at once.

One attention computation forces every token to ask one question per layer. The fix is to run several in parallel, each on its own slice of the embedding. Each slice is called a head and learns its own query/key/value matrices. One head ends up tracking word order; another tracks which subject a verb belongs to; another binds adjectives to their nouns. The heads' outputs concatenate and pass through one final projection.

A transformer block: attention plus a small per-token network.

A transformer stacks identical blocks. One block does two things in order: multi-head attention, so tokens mix information across the sequence, then a small two-layer network (a feed-forward or MLP) applied to each token independently — this is where most parameters live. Residual connections let information skip past each sublayer; layer norm keeps activation sizes stable as the stack goes deep.

Stack 18 of these blocks and you have the language-model half of PaliGemma — the backbone π₀.5 starts from. ²

The mask: who is allowed to look at whom.

Attention defaults to letting every token see every other. For a model generating text one token at a time, that is too much — a token must not peek at words it has not produced yet. The fix is a mask: a 2D table the same size as the attention matrix, with forbidden cells set to $-\infty$ before the softmax (which forces their weights to zero).

π₀.5 uses a pattern called prefix-LM. The input — image patches plus the language prompt — is the prefix; tokens inside the prefix see each other in both directions because the model is reading them, not generating them. Anything after the prefix — the suffix — is causal: each new token sees all the prefix and only the earlier suffix tokens.

That is the substrate. Tokens are vectors; attention is a learned weighted average; a block is attention plus a per-token network with residuals; stack the blocks under the prefix-LM mask and you have the vision-language model π₀.5 is built on.

The next page introduces the action expert — a second transformer that talks to this one and produces continuous robot actions instead of words.