I have started building tokenscope and here’s the idea:
tokenscope is an offline terminal tokenizer explorer for local HuggingFace tokenizer files. It loads tokenizer folders from disk, lets you type text interactively, and shows colored token spans, token IDs, vocabulary stats, token inspection, decode round-trip checks, special-token metadata, prompt budgets, corpus analysis, BPE merge reconstruction, vocabulary search, and optional side-by-side tokenizer comparison.