LearnFrenzy

Sign in Take a free test

LearnFrenzy

Menu

Home

Practice

Build & Compete

Tests

Interview

Learn

Support

Contact Us

Free mock tests

Timed tests with instant scores — no sign-up needed.

Glossary · Artificial Intelligence

What is Token?

A small chunk of text — a word, part of a word or punctuation — that a language model processes as one unit.

In more detail

Model limits and pricing are counted in tokens. In English, 100 tokens is roughly 75 words.

Example

"unbelievable" may be split into "un", "believ", "able".

Learn it properlyHow Large Language Models (LLMs) Work — Without the Maths11 min lesson · Artificial Intelligence

Related terms

AI AgentAn AI system that can plan steps and use tools — search, code, apps — to complete a task with less step-by-step instruction.Artificial Intelligence (AI)Software that learns patterns from data and uses them to make predictions, decisions or new content.Bias (in AI)Systematic unfairness in AI output, usually inherited from imbalanced or historical training data.Context WindowThe maximum amount of text a language model can consider at one time.Deep LearningMachine learning that uses neural networks with many layers, especially good for images, audio and language.Fine-TuningFurther training a pre-trained model on a smaller, specific dataset to adapt it to a task or style.