From Text to Tokens: WordPiece & Input Representations

Before any loss is computed: WordPiece splits, the sum of token + segment + position embeddings, and the [CLS]/[SEP] plumbing.

Playlist by @bert_pretraining_turn

9 tracks, shared on Audicious.

1. WordPiece Tokenization — Hugging Face
2. LLM Tokenizers Explained: BPE Encoding, WordPiece and SentencePiece — DataMListic
3. WordPiece Tokenization in NLP — TechViz - The Data Science Guy
4. Byte Pair Encoding Tokenization — Hugging Face
5. Understanding BERT Embeddings and Tokenization | NLP | HuggingFace| Data Science | Machine Learning — Rohan-Paul-AI
6. BERT Research - Ep. 2 - WordPiece Embeddings — InnerWorkingsAI
7. What makes LLM tokenizers different from each other? GPT4 vs. FlanT5 Vs. Starcoder Vs. BERT and more — Jay Alammar
8. How to Build a Bert WordPiece Tokenizer in Python and HuggingFace — James Briggs
9. Let's build the GPT Tokenizer — Andrej Karpathy