# How AI turns a sentence into an image

> A process map exported from forks.world (https://forks.world). Author: Brian. Source: https://forks.world/p/cmv0kpqkw003tjn064o6q1msp
> It describes who does what, in what order, and with which technology. Read the plain-language sections first; the JSON at the end is the exact, complete data.

## Summary

- 17 steps across 2 layers
- Persona: You
- Technology / systems: Text encoder, Random noise, Denoising model, Safety filter

## Pain points

Where this process hurts, worst first:

- **Are the hands right?** (How AI turns a sentence into an image; annoying): Hands, text and counting objects are still famous weak spots

## How to read this file

- A process is made of **layers**. The main layer is the top level; any step of kind `subprocess` opens a detail layer with its own lanes and steps.
- **Lanes** are either persona (people/roles) or technology (systems). Each step belongs to one lane; `using` lists tools a person uses for that step.
- Step kinds: `start`, `task`, `decision` (its arrows carry the answer as a label), `subprocess`, `end`.

## How AI turns a sentence into an image

Lanes:
- You: person / role (id `you`)
- Text encoder: technology / system (id `enc`)
- Random noise: technology / system (id `noise`)
- Denoising model: technology / system (id `model`)
- Safety filter: technology / system (id `filter`)

Steps, in the order they run:

1. **Write a prompt** [start; You; id `d1`]
   - then Turn the words into a meaning vector
2. **Turn the words into a meaning vector** [task; Text encoder; id `d2`]
   - then Start with a screen of pure random noise
3. **Start with a screen of pure random noise** [task; Random noise; id `d3`]
   - then Predict which noise to remove, guided by the prompt
4. **Predict which noise to remove, guided by the prompt** [subprocess; Denoising model; id `d4`], opens the detail layer "Noise to picture"
   - then Remove a little noise
5. **Remove a little noise** [task; Denoising model; id `d5`]
   - then Is it clear enough?
6. **Is it clear enough?** [decision; Denoising model; id `d6`]
   - if "Yes" go to Check for disallowed content; if "No — another step" go to Predict which noise to remove, guided by the prompt
7. **Check for disallowed content** [task; Safety filter; id `d7`]
   - then Is it allowed?
8. **Is it allowed?** [decision; Safety filter; id `d8`]
   - if "Yes" go to You get an image; if "No" go to Block or blur it
9. **You get an image** [task; You; id `d9`]
   - then Are the hands right?
10. **Are the hands right?** [decision; You; id `d11`] ⚠ PAIN POINT (annoying): Hands, text and counting objects are still famous weak spots
   - if "Yes" go to Done; if "No" go to Rewrite the prompt and try again
11. **Done** [end; You; id `d12`]
12. **Rewrite the prompt and try again** [task; You; id `d13`]
   - then Write a prompt
13. **Block or blur it** [end; Safety filter; id `d10`]

```mermaid
flowchart LR
  subgraph l_you["You"]
    n_d1(["Write a prompt"])
    n_d9["You get an image"]
    n_d11{"Are the hands right?"}
    n_d13["Rewrite the prompt and try again"]
    n_d12(["Done"])
  end
  subgraph l_enc["Text encoder"]
    n_d2["Turn the words into a meaning vector"]
  end
  subgraph l_noise["Random noise"]
    n_d3["Start with a screen of pure random noise"]
  end
  subgraph l_model["Denoising model"]
    n_d4[["Predict which noise to remove, guided by the prompt"]]
    n_d5["Remove a little noise"]
    n_d6{"Is it clear enough?"}
  end
  subgraph l_filter["Safety filter"]
    n_d7["Check for disallowed content"]
    n_d8{"Is it allowed?"}
    n_d10(["Block or blur it"])
  end
  n_d1 --> n_d2
  n_d2 --> n_d3
  n_d3 --> n_d4
  n_d4 --> n_d5
  n_d5 --> n_d6
  n_d6 -->|"Yes"| n_d7
  n_d6 -->|"No — another step"| n_d4
  n_d7 --> n_d8
  n_d8 -->|"Yes"| n_d9
  n_d8 -->|"No"| n_d10
  n_d9 --> n_d11
  n_d11 -->|"Yes"| n_d12
  n_d11 -->|"No"| n_d13
  n_d13 --> n_d1
```

### Noise to picture (detail of step "Predict which noise to remove, guided by the prompt")

Lanes:
- Denoising model: technology / system (id `model`)

Steps, in the order they run:

1. **The image is mostly static** [start; Denoising model; id `n1`]
   - then Shapes begin to appear
2. **Shapes begin to appear** [task; Denoising model; id `n2`]
   - then Details sharpen
3. **Details sharpen** [task; Denoising model; id `n3`]
   - then The final clean image
4. **The final clean image** [end; Denoising model; id `n4`]

```mermaid
flowchart LR
  subgraph l_model["Denoising model"]
    n_n1(["The image is mostly static"])
    n_n2["Shapes begin to appear"]
    n_n3["Details sharpen"]
    n_n4(["The final clean image"])
  end
  n_n1 --> n_n2
  n_n2 --> n_n3
  n_n3 --> n_n4
```

## Full data (JSON)

```json
{
  "title": "How AI turns a sentence into an image",
  "maps": {
    "m_root": {
      "id": "m_root",
      "lanes": [
        {
          "id": "you",
          "name": "You",
          "type": "person"
        },
        {
          "id": "enc",
          "name": "Text encoder",
          "type": "system"
        },
        {
          "id": "noise",
          "name": "Random noise",
          "type": "system"
        },
        {
          "id": "model",
          "name": "Denoising model",
          "type": "system"
        },
        {
          "id": "filter",
          "name": "Safety filter",
          "type": "system"
        }
      ],
      "steps": [
        {
          "id": "d1",
          "kind": "start",
          "lane": "you",
          "next": [
            {
              "to": "d2"
            }
          ],
          "uses": [],
          "label": "Write a prompt"
        },
        {
          "id": "d2",
          "kind": "task",
          "lane": "enc",
          "next": [
            {
              "to": "d3"
            }
          ],
          "uses": [],
          "label": "Turn the words into a meaning vector"
        },
        {
          "id": "d3",
          "kind": "task",
          "lane": "noise",
          "next": [
            {
              "to": "d4"
            }
          ],
          "uses": [],
          "label": "Start with a screen of pure random noise"
        },
        {
          "id": "d4",
          "kind": "subprocess",
          "lane": "model",
          "next": [
            {
              "to": "d5"
            }
          ],
          "uses": [],
          "child": "m_denoise",
          "label": "Predict which noise to remove, guided by the prompt"
        },
        {
          "id": "d5",
          "kind": "task",
          "lane": "model",
          "next": [
            {
              "to": "d6"
            }
          ],
          "uses": [],
          "label": "Remove a little noise"
        },
        {
          "id": "d6",
          "kind": "decision",
          "lane": "model",
          "next": [
            {
              "to": "d7",
              "label": "Yes"
            },
            {
              "to": "d4",
              "label": "No — another step"
            }
          ],
          "uses": [],
          "label": "Is it clear enough?"
        },
        {
          "id": "d7",
          "kind": "task",
          "lane": "filter",
          "next": [
            {
              "to": "d8"
            }
          ],
          "uses": [],
          "label": "Check for disallowed content"
        },
        {
          "id": "d8",
          "kind": "decision",
          "lane": "filter",
          "next": [
            {
              "to": "d9",
              "label": "Yes"
            },
            {
              "to": "d10",
              "label": "No"
            }
          ],
          "uses": [],
          "label": "Is it allowed?"
        },
        {
          "id": "d10",
          "kind": "end",
          "lane": "filter",
          "next": [],
          "uses": [],
          "label": "Block or blur it"
        },
        {
          "id": "d9",
          "kind": "task",
          "lane": "you",
          "next": [
            {
              "to": "d11"
            }
          ],
          "uses": [],
          "label": "You get an image"
        },
        {
          "id": "d11",
          "kind": "decision",
          "lane": "you",
          "next": [
            {
              "to": "d12",
              "label": "Yes"
            },
            {
              "to": "d13",
              "label": "No"
            }
          ],
          "pain": {
            "note": "Hands, text and counting objects are still famous weak spots",
            "level": 1
          },
          "uses": [],
          "label": "Are the hands right?"
        },
        {
          "id": "d13",
          "kind": "task",
          "lane": "you",
          "next": [
            {
              "to": "d1"
            }
          ],
          "uses": [],
          "label": "Rewrite the prompt and try again"
        },
        {
          "id": "d12",
          "kind": "end",
          "lane": "you",
          "next": [],
          "uses": [],
          "label": "Done"
        }
      ],
      "title": "How AI turns a sentence into an image",
      "parent": null
    },
    "m_denoise": {
      "id": "m_denoise",
      "lanes": [
        {
          "id": "model",
          "name": "Denoising model",
          "type": "system"
        }
      ],
      "steps": [
        {
          "id": "n1",
          "kind": "start",
          "lane": "model",
          "next": [
            {
              "to": "n2"
            }
          ],
          "uses": [],
          "label": "The image is mostly static"
        },
        {
          "id": "n2",
          "kind": "task",
          "lane": "model",
          "next": [
            {
              "to": "n3"
            }
          ],
          "uses": [],
          "label": "Shapes begin to appear"
        },
        {
          "id": "n3",
          "kind": "task",
          "lane": "model",
          "next": [
            {
              "to": "n4"
            }
          ],
          "uses": [],
          "label": "Details sharpen"
        },
        {
          "id": "n4",
          "kind": "end",
          "lane": "model",
          "next": [],
          "uses": [],
          "label": "The final clean image"
        }
      ],
      "title": "Noise to picture",
      "parent": {
        "map": "m_root",
        "step": "d4"
      }
    }
  }
}
```