Lemmatisation

Reducing words to their dictionary form, such as 'be' for is and was, so variants count as one token. More accurate than stemming, and more costly to run.

Where you will meet it

Lemmatisation appears in 2 learning outcomes on the CFA syllabus. A few of them:

• Explain techniques for extracting, choosing and engineering features from text data
  CFA Level II · Quantitative Methods

  “Tokenisation splits it into words; normalisation lowercases, removes stop words, and applies stemming or lemmatisation.”

• Explain how text-based data is prepared, wrangled and explored for use in financial forecasting
  CFA Level II · Quantitative Methods

  “Preprocessing then tokenises and normalises: lowercasing, removing stop words, and stemming (cutting to a root such as "downgrad") or lemmatisation (mapping to a dictionary word).”

See also

Stemming · Token

Practise it

• Today's practice question One question, with every wrong option explained.
• Free Level I diagnostic test Twelve questions that show which topics to start with.
• Free finance calculators Check your arithmetic on the exam's core formulas.

All terms beginning with L · Glossary