tokenscope is an offline terminal tokenizer explorer for local HuggingFace tokenizer files. It loads tokenizer folders from disk, lets you type text interactively, and shows colored token spans, token IDs, vocabulary stats, token inspection, decode round-trip checks, special-token metadata, prompt budgets, corpus analysis, BPE merge reconstruction, vocabulary search, and optional side-by-side tokenizer comparison.
tokenscope v0.2.0 is the second public release and the first release focused on making the app useful outside a single interactive local-tokenizer workflow. v0.1.0 established the terminal UI, tokenizer inspection, exports, comparison mode, and the binary release pipeline. v0.2.0 expands that base into a more scriptable, configurable, and measurable tool.
Release goals
The main goal for v0.2.0 was to remove friction around repeated tokenizer analysis. The app already had a broad inspection surface, but common workflows still required too much manual setup: choosing the same tokenizer every time, keeping budget defaults in memory, copying token IDs manually, benchmarking outside the app, or writing small wrapper scripts around exports.
This release focused on loading common tokenizers more quickly, making repeated analysis reproducible through config files, supporting CLI and CI workflows with stdin input, adding benchmark visibility inside the TUI, surfacing budget pressure earlier, and keeping release notes, docs, and build outputs organized for future releases.
Optional HuggingFace Hub loading
v0.2.0 adds explicit Hub loading through –hub, default_hub_model, and the TUI folder browser’s Load from Hub controls. This is intentionally opt-in. tokenscope still stays local-first and does not make runtime network calls unless the user requests a Hub tokenizer.
The implementation routes Hub tokenizers through the same tokenizer engine abstraction as local tokenizers, which keeps downstream analysis, comparison, exports, and metadata behavior consistent.
Configuration files
The new config.py module adds .tokenscoperc and tokenscope.json support. Config files can define defaults for tokenizer paths, Hub model IDs, compare tokenizers, corpus and batch paths, export format, budget, Hub cache location, and special-token encoding.
The init-config subcommand writes a starter config file, which makes it easier to standardize a workspace without memorizing every CLI flag.
Headless stdin analysis
Headless analyze mode can now read from stdin with –stdin, so shell pipelines and CI scripts can pass prompts directly into tokenscope.
Example command:
cat prompt.txt | python main.py analyze –tokenizer ./gpt2-local –stdin –budget 8192
The headless path also now honors config defaults when CLI arguments are omitted, which makes automated use less repetitive.
Benchmark tab
The new Benchmark tab measures tokenization throughput for the active input and reports tokens per second and characters per second. In compare mode, the benchmark can show primary-vs-compare speed deltas in the same interface where token boundaries and IDs are already being inspected.
This is meant as a practical local measurement, not a lab-grade benchmark suite. It is useful for quickly spotting large tokenizer performance differences while testing real prompts.
Budget alerts
Budget usage is now more visible. The header includes an ambient budget badge, and the Stats panel shows a dedicated budget row. Both update based on utilization thresholds: green below 80 percent, yellow from 80 to 95 percent, red above 95 percent, and bold red above 100 percent.
The intent is to make over-budget input obvious without forcing users to keep the Budget tab open.
Clipboard support
Ctrl+Y copies the current primary token IDs to the clipboard as a space-separated list. This is a small feature, but it removes an annoying manual step when moving token IDs into tests, issue reports, or comparison notes.
Cost profile persistence
The Cost tab now supports persistent user-defined pricing profiles. Project state was moved to schema version 2 so pricing profile information can round-trip with saved projects while still loading older v1 project files safely.
tokenscope v0.1 is the first public release of the project: an offline terminal app for inspecting local HuggingFace tokenizer files without pulling in transformers or making runtime network calls.
The goal for this release was to make tokenization visible. Tokenizers are often treated like a black box, but small differences in splits, offsets, special tokens, byte handling, and chat formatting can change cost, context usage, retrieval behavior, and model output quality. v0.1 turns those details into something you can inspect from a terminal.
The core TUI now loads local tokenizer folders, direct tokenizer.json files, WordPiece vocab files, and BPE vocab/merge layouts. Once loaded, the app shows colored token spans, token IDs, offsets, byte representations, text slices, vocabulary stats, and per-token details.
The selected-token inspector was one of the most useful additions. It tracks the current token, supports keyboard navigation with [ and ], syncs with token-table row selection, and shows the token text, ID, offsets, bytes, display width, local frequency, vocabulary membership, and a best-effort BPE merge tree.
Compare mode became a major part of the release. Two tokenizers can now be loaded side by side against the same input, with aligned token views, token-count deltas, compression deltas, boundary differences, token differences, ID differences, corpus comparison, and tokenizer metadata inspection.
The analysis surface grew beyond single prompts. v0.1 includes corpus analysis, batch prompt analysis, prompt budget checks, chat-template budget rendering, tokenizer pipeline debugging, tokenizer diffing, prompt packing simulation, regression suites, Unicode inspection, RAG chunk analysis, distribution summaries, cost estimates, and tokenizer repair suggestions.
Exports were expanded to JSON, CSV, Markdown, and HTML. The headless analyze command makes the same analysis useful in scripts and CI, including budget and regression failure modes.
I have started building tokenscope and here’s the idea:
tokenscope is an offline terminal tokenizer explorer for local HuggingFace tokenizer files. It loads tokenizer folders from disk, lets you type text interactively, and shows colored token spans, token IDs, vocabulary stats, token inspection, decode round-trip checks, special-token metadata, prompt budgets, corpus analysis, BPE merge reconstruction, vocabulary search, and optional side-by-side tokenizer comparison.
