Token
The unit a language model reads and writes: a word or a piece of one. The model generates text by estimating probabilities for the next token, picking one and repeating.
Where you will meet it
Token appears in 5 learning outcomes on the CFA syllabus. A few of them:
- Explain how training works for large language models and the way they produce text
“Text is split into tokens, which are words or pieces of words.”
- Explain what digital assets are and their features as investments
“Digital assets include cryptocurrencies intended as a medium of exchange or store of value, tokens representing a claim on a project or service, and stablecoins pegged to a currency or asset.”
- Set out the aims, techniques and examples of exploring data
“Exploratory data analysis summarises and visualises the data: histograms, bar charts and box plots for one feature, scatterplots and correlation matrices for pairs, and word clouds for the most frequent text tokens.”
- Explain techniques for extracting, choosing and engineering features from text data
“A bag of words records which tokens occur, ignoring order, and a document term matrix has one row per document and one column per token.”
Practise it
Practise token until it is automatic: questions that explain every wrong option.
Take the free diagnostic →