tokenscope v0.2.0 is the second public release and the first release focused on making the app useful outside a single interactive local-tokenizer workflow. v0.1.0 established the terminal UI, tokenizer inspection, exports, comparison mode, and the binary release pipeline. v0.2.0 expands that base into a more scriptable, configurable, and measurable tool.

Release goals

The main goal for v0.2.0 was to remove friction around repeated tokenizer analysis. The app already had a broad inspection surface, but common workflows still required too much manual setup: choosing the same tokenizer every time, keeping budget defaults in memory, copying token IDs manually, benchmarking outside the app, or writing small wrapper scripts around exports.

This release focused on loading common tokenizers more quickly, making repeated analysis reproducible through config files, supporting CLI and CI workflows with stdin input, adding benchmark visibility inside the TUI, surfacing budget pressure earlier, and keeping release notes, docs, and build outputs organized for future releases.

Optional HuggingFace Hub loading

v0.2.0 adds explicit Hub loading through –hub, default_hub_model, and the TUI folder browser’s Load from Hub controls. This is intentionally opt-in. tokenscope still stays local-first and does not make runtime network calls unless the user requests a Hub tokenizer.

The implementation routes Hub tokenizers through the same tokenizer engine abstraction as local tokenizers, which keeps downstream analysis, comparison, exports, and metadata behavior consistent.

Configuration files

The new config.py module adds .tokenscoperc and tokenscope.json support. Config files can define defaults for tokenizer paths, Hub model IDs, compare tokenizers, corpus and batch paths, export format, budget, Hub cache location, and special-token encoding.

The init-config subcommand writes a starter config file, which makes it easier to standardize a workspace without memorizing every CLI flag.

Headless stdin analysis

Headless analyze mode can now read from stdin with –stdin, so shell pipelines and CI scripts can pass prompts directly into tokenscope.

Example command:

cat prompt.txt | python main.py analyze –tokenizer ./gpt2-local –stdin –budget 8192

The headless path also now honors config defaults when CLI arguments are omitted, which makes automated use less repetitive.

Benchmark tab

The new Benchmark tab measures tokenization throughput for the active input and reports tokens per second and characters per second. In compare mode, the benchmark can show primary-vs-compare speed deltas in the same interface where token boundaries and IDs are already being inspected.

This is meant as a practical local measurement, not a lab-grade benchmark suite. It is useful for quickly spotting large tokenizer performance differences while testing real prompts.

Budget alerts

Budget usage is now more visible. The header includes an ambient budget badge, and the Stats panel shows a dedicated budget row. Both update based on utilization thresholds: green below 80 percent, yellow from 80 to 95 percent, red above 95 percent, and bold red above 100 percent.

The intent is to make over-budget input obvious without forcing users to keep the Budget tab open.

Clipboard support

Ctrl+Y copies the current primary token IDs to the clipboard as a space-separated list. This is a small feature, but it removes an annoying manual step when moving token IDs into tests, issue reports, or comparison notes.

Cost profile persistence

The Cost tab now supports persistent user-defined pricing profiles. Project state was moved to schema version 2 so pricing profile information can round-trip with saved projects while still loading older v1 project files safely.