Token
isation
BPE
How a language model turns your words into the numbers it actually reads
Concept
The Words
Subwords
Token IDs
Sequence
◀ PREV
1 / 9
▶ AUTO
NEXT ▶
←
→
navigate |
Space
auto-play |
R
restart