Tokenization is a fundamental concept for large language models, but what exactly is a token, how do tokenizers work, and why would we want to use tokens in the first place?

Join this session and we will unravel the mechanisms behind transforming textual data into machine-understandable formats together.

Through real-world examples and demos, you will grasp the essence of tokenization, the pitfalls, the relevance to prompt engineering, and why it is important to have some understanding of these fundamental building blocks of large language models.