GigaToken: ~1000x faster Language model tokenization

GigaToken is a new, high-performance language model tokenizer that claims to be up to 1000x faster than existing solutions like HuggingFace. It is designed as a drop-in replacement that supports multiple hardware configurations and common tokenizer formats.