# How a language model writes one word at a time

> A process map exported from forks.world (https://forks.world). Author: Brian. Source: https://forks.world/p/cmv0kpq1p003ojn06v04h6iiq
> It describes who does what, in what order, and with which technology. Read the plain-language sections first; the JSON at the end is the exact, complete data.

## Summary

- 13 steps across 2 layers
- Persona: You
- Technology / systems: The tokenizer, The neural network, The sampler

## How to read this file

- A process is made of **layers**. The main layer is the top level; any step of kind `subprocess` opens a detail layer with its own lanes and steps.
- **Lanes** are either persona (people/roles) or technology (systems). Each step belongs to one lane; `using` lists tools a person uses for that step.
- Step kinds: `start`, `task`, `decision` (its arrows carry the answer as a label), `subprocess`, `end`.

## How a language model writes one word at a time

Lanes:
- You: person / role (id `you`)
- The tokenizer: technology / system (id `tok`)
- The neural network: technology / system (id `net`)
- The sampler: technology / system (id `sample`)

Steps, in the order they run:

1. **You type a prompt** [start; You; id `a1`]
   - then Split it into tokens (word pieces)
2. **Split it into tokens (word pieces)** [task; The tokenizer; id `a2`]
   - then Turn the tokens into numbers (embeddings)
3. **Turn the tokens into numbers (embeddings)** [task; The neural network; id `a3`]
   - then Attention layers decide which earlier words matter most
4. **Attention layers decide which earlier words matter most** [subprocess; The neural network; id `a4`], opens the detail layer "Attention"
   - then Compute a probability for every possible next token
5. **Compute a probability for every possible next token** [task; The neural network; id `a5`]
   - then Pick one (temperature controls how adventurous)
6. **Pick one (temperature controls how adventurous)** [task; The sampler; id `a6`]
   - then Is it a stop token?
7. **Is it a stop token?** [decision; You; id `a7`]
   - if "Yes" go to The answer is complete; if "No" go to Add the token to the text and repeat
8. **The answer is complete** [end; You; id `a8`]
9. **Add the token to the text and repeat** [task; The neural network; id `a9`]
   - then Turn the tokens into numbers (embeddings)

```mermaid
flowchart LR
  subgraph l_you["You"]
    n_a1(["You type a prompt"])
    n_a7{"Is it a stop token?"}
    n_a8(["The answer is complete"])
  end
  subgraph l_tok["The tokenizer"]
    n_a2["Split it into tokens (word pieces)"]
  end
  subgraph l_net["The neural network"]
    n_a3["Turn the tokens into numbers (embeddings)"]
    n_a4[["Attention layers decide which earlier words matter most"]]
    n_a5["Compute a probability for every possible next token"]
    n_a9["Add the token to the text and repeat"]
  end
  subgraph l_sample["The sampler"]
    n_a6["Pick one (temperature controls how adventurous)"]
  end
  n_a1 --> n_a2
  n_a2 --> n_a3
  n_a3 --> n_a4
  n_a4 --> n_a5
  n_a5 --> n_a6
  n_a6 --> n_a7
  n_a7 -->|"Yes"| n_a8
  n_a7 -->|"No"| n_a9
  n_a9 --> n_a3
```

### Attention (detail of step "Attention layers decide which earlier words matter most")

Lanes:
- The neural network: technology / system (id `net`)

Steps, in the order they run:

1. **For each word make a query, a key and a value** [start; The neural network; id `x1`]
   - then Compare queries with keys: how related are these words?
2. **Compare queries with keys: how related are these words?** [task; The neural network; id `x2`]
   - then Weight the values by relevance
3. **Weight the values by relevance** [task; The neural network; id `x3`]
   - then Repeat in many layers and many "heads"
4. **Repeat in many layers and many "heads"** [end; The neural network; id `x4`]

```mermaid
flowchart LR
  subgraph l_net["The neural network"]
    n_x1(["For each word make a query, a key and a value"])
    n_x2["Compare queries with keys: how related are these words?"]
    n_x3["Weight the values by relevance"]
    n_x4(["Repeat in many layers and many 'heads'"])
  end
  n_x1 --> n_x2
  n_x2 --> n_x3
  n_x3 --> n_x4
```

## Full data (JSON)

```json
{
  "title": "How a language model writes one word at a time",
  "maps": {
    "m_attn": {
      "id": "m_attn",
      "lanes": [
        {
          "id": "net",
          "name": "The neural network",
          "type": "system"
        }
      ],
      "steps": [
        {
          "id": "x1",
          "kind": "start",
          "lane": "net",
          "next": [
            {
              "to": "x2"
            }
          ],
          "uses": [],
          "label": "For each word make a query, a key and a value"
        },
        {
          "id": "x2",
          "kind": "task",
          "lane": "net",
          "next": [
            {
              "to": "x3"
            }
          ],
          "uses": [],
          "label": "Compare queries with keys: how related are these words?"
        },
        {
          "id": "x3",
          "kind": "task",
          "lane": "net",
          "next": [
            {
              "to": "x4"
            }
          ],
          "uses": [],
          "label": "Weight the values by relevance"
        },
        {
          "id": "x4",
          "kind": "end",
          "lane": "net",
          "next": [],
          "uses": [],
          "label": "Repeat in many layers and many \"heads\""
        }
      ],
      "title": "Attention",
      "parent": {
        "map": "m_root",
        "step": "a4"
      }
    },
    "m_root": {
      "id": "m_root",
      "lanes": [
        {
          "id": "you",
          "name": "You",
          "type": "person"
        },
        {
          "id": "tok",
          "name": "The tokenizer",
          "type": "system"
        },
        {
          "id": "net",
          "name": "The neural network",
          "type": "system"
        },
        {
          "id": "sample",
          "name": "The sampler",
          "type": "system"
        }
      ],
      "steps": [
        {
          "id": "a1",
          "kind": "start",
          "lane": "you",
          "next": [
            {
              "to": "a2"
            }
          ],
          "uses": [],
          "label": "You type a prompt"
        },
        {
          "id": "a2",
          "kind": "task",
          "lane": "tok",
          "next": [
            {
              "to": "a3"
            }
          ],
          "uses": [],
          "label": "Split it into tokens (word pieces)"
        },
        {
          "id": "a3",
          "kind": "task",
          "lane": "net",
          "next": [
            {
              "to": "a4"
            }
          ],
          "uses": [],
          "label": "Turn the tokens into numbers (embeddings)"
        },
        {
          "id": "a4",
          "kind": "subprocess",
          "lane": "net",
          "next": [
            {
              "to": "a5"
            }
          ],
          "uses": [],
          "child": "m_attn",
          "label": "Attention layers decide which earlier words matter most"
        },
        {
          "id": "a5",
          "kind": "task",
          "lane": "net",
          "next": [
            {
              "to": "a6"
            }
          ],
          "uses": [],
          "label": "Compute a probability for every possible next token"
        },
        {
          "id": "a6",
          "kind": "task",
          "lane": "sample",
          "next": [
            {
              "to": "a7"
            }
          ],
          "uses": [],
          "label": "Pick one (temperature controls how adventurous)"
        },
        {
          "id": "a7",
          "kind": "decision",
          "lane": "you",
          "next": [
            {
              "to": "a8",
              "label": "Yes"
            },
            {
              "to": "a9",
              "label": "No"
            }
          ],
          "uses": [],
          "label": "Is it a stop token?"
        },
        {
          "id": "a9",
          "kind": "task",
          "lane": "net",
          "next": [
            {
              "to": "a3"
            }
          ],
          "uses": [],
          "label": "Add the token to the text and repeat"
        },
        {
          "id": "a8",
          "kind": "end",
          "lane": "you",
          "next": [],
          "uses": [],
          "label": "The answer is complete"
        }
      ],
      "title": "How a language model writes one word at a time",
      "parent": null
    }
  }
}
```