tokenscope v0.1 is the first public release of the project: an offline terminal app for inspecting local HuggingFace tokenizer files without pulling in transformers or making runtime network calls.

The goal for this release was to make tokenization visible. Tokenizers are often treated like a black box, but small differences in splits, offsets, special tokens, byte handling, and chat formatting can change cost, context usage, retrieval behavior, and model output quality. v0.1 turns those details into something you can inspect from a terminal.

What Shipped

The core TUI now loads local tokenizer folders, direct tokenizer.json files, WordPiece vocab files, and BPE vocab/merge layouts. Once loaded, the app shows colored token spans, token IDs, offsets, byte representations, text slices, vocabulary stats, and per-token details.

The selected-token inspector was one of the most useful additions. It tracks the current token, supports keyboard navigation with [ and ], syncs with token-table row selection, and shows the token text, ID, offsets, bytes, display width, local frequency, vocabulary membership, and a best-effort BPE merge tree.

Compare mode became a major part of the release. Two tokenizers can now be loaded side by side against the same input, with aligned token views, token-count deltas, compression deltas, boundary differences, token differences, ID differences, corpus comparison, and tokenizer metadata inspection.

The analysis surface grew beyond single prompts. v0.1 includes corpus analysis, batch prompt analysis, prompt budget checks, chat-template budget rendering, tokenizer pipeline debugging, tokenizer diffing, prompt packing simulation, regression suites, Unicode inspection, RAG chunk analysis, distribution summaries, cost estimates, and tokenizer repair suggestions.

Exports were expanded to JSON, CSV, Markdown, and HTML. The headless analyze command makes the same analysis useful in scripts and CI, including budget and regression failure modes.