The process of breaking down text into smaller units, called tokens, which can be as short as a character or as long as a word or phrase, for processing by AI models.
Last updated 8 months ago