AN EXPERIMENT IN MACHINE WRITING

Automatic Writing.

Inspired by the Surrealists’ practice of automatic writing, this experiment lets language unfold without a written prompt. A language model uses recent tokens to predict what could come next, then selects the next token by sampling from those probabilities. A 3D view tracks attention weights from a selected layer and maps tokens through their embeddings.

The model has no unconscious mind to uncover. This demo offers a look under the hood of how large language models generate text through learned patterns, attention, and probabilistic sampling within a limited context window.

Model

What you will see: tokens appear in a 3D space as the model writes, with the probabilities and attention behind each choice. Small base models work best. Loading includes preparing a fixed 3D embedding projection. Models are preloaded in 16-bit precision (Gemma in BF16 to keep generation numerically stable), up to 8 GB; each visitor has a separate stream. Generation uses short GPU sessions; Pause releases the GPU.