The Long Run
← Back to AI & Data Science

AI & Data Science

LLM & Tokenisation

A short series on how large language models actually read and write text — one part at a time.

Part 1Tokens 101: The Secret Language Your AI Actually SpeaksA beginner-friendly introduction to what tokens are, how they differ from words and characters, and why they matter when using LLMs like ChatGPT and Claude.Part 2Inside the Tokeniser: BPE, WordPiece, SentencePiece & Unigram ExplainedAn example-driven walkthrough of the four major tokenisation algorithms behind modern LLMs, with a step-by-step explanation.Part 3Token Limits: Why Your AI Chatbot Suddenly Forgets EverythingA deep dive into how token limits affect the performance of large language models and what you can do about it.Part 4Context Windows Demystified: How Much Can Your AI Actually Remember?The finale of our LLM tokenization series: what context windows are, how they relate to token limits, and how real AI products manage them.