Show HN: TriusLD – A Small Tokenizer for LLMs with Robust Multilingual Support
Building a 64,972-token byte-level BPE that covers 19 writing systems without giving up English: per-script quotas, character coverage, and what went wrong on the way.
Building a 64,972-token byte-level BPE that covers 19 writing systems without giving up English: per-script quotas, character coverage, and what went wrong on the way.