The Building Blocks Hidden in A.I.: How Tokens Work
This report examines the fundamental mechanics of artificial intelligence, specifically focusing on tokenisation.
The story
in brief.
What happened, who it affects, and why it landed on our intelligence desk — in plain English, sixty seconds or less.
This report examines the fundamental mechanics of artificial intelligence, specifically focusing on tokenisation. It explains how large language models process text by breaking it into discrete units known as tokens. The article clarifies that these tokens are not merely individual words but can represent sub-words or characters, influencing how AI interprets context and generates responses. Understanding this underlying structure is crucial for demystifying AI capabilities and limitations. The piece provides a technical yet accessible overview, helping professionals grasp why certain prompts yield specific results and how the model’s architecture dictates its operational boundaries in practical applications.
Indicative index of skill mentions in professional job postings for the field this story moves (2019 = 1)
What this means
for your career.
Every headline redistributes opportunity. Here is who this one rewards — and how to be on the right side of it.
Understanding tokenisation transforms how you interact with AI tools, moving you from passive user to strategic operator. You must now prioritise prompt engineering and technical literacy over vague intuition. Professionals in content creation, data analysis, and software development should pay close attention, as misinterpreting token limits can lead to costly errors or inefficient workflows. To stay competitive, you should experiment with prompt structures to see how tokenisation affects output quality. Invest time in learning the technical constraints of the models you use daily. This knowledge allows you to optimise costs, improve accuracy, and communicate more effectively with technical teams, ensuring you leverage AI as a precise instrument rather than a black box.
For this story, the fields that convert it into pay are Artificial Intelligence, AI Strategy, Data Science and Requirements Engineering — exactly the programmes matched below. The chart shows the indicative salary uplift certified professionals report in each of those fields; every one of them is a click away, from £79, finishing in as few as 20 days.
Indicative ranges from published continuing-education salary studies, by field — each field is a recommended course below
Enrol in these
to benefit.
These programmes position you for exactly this shift — online, self-paced, and finishing with a verifiable certificate you can share the day you pass.
Best match
Certificate in Artificial Intelligence
Recommended
Certificate in AI Strategy
Recommended
Certificate in Data Science
Recommended
Certificate in Requirements Engineering
Launch pricing shown against standard certificate value. The fee you see today is the fee you lock in.
This briefing is based on reporting by The New York Times on 9 Sep, 09:00. Read the original coverage →
Read the news. Then outrun it.
Every OnlineUni programme is online, self-paced and from £79 — so a headline today can be your credential this month.