centerpixel.com
← tools

tokdump

Like hexdump, but for tokens.

Dumps token IDs in the three-column layout hexdump(1) has used since 1978, so you can see exactly where a string splits — and why an emoji costs you four tokens.

tokdumptokdump showing where a string splits into tokens.tokdump$printf 'Hello, world!' | tokdump0000000 13225 11 2375 0 Hello|,|·world|!0000004$printf '💯🔥' | tokdump -n0000000 31446 107 \xf0\x9f\x92|\xaf0000002 96606 🔥0000003$Three columns: offset, token IDs, and thetext they cover. The layout hexdump has usedsince 1978.Two emoji. The first costs two tokens andsplits mid-character; the second is a singletoken.Narrow: two token IDs per row instead offour.
Hover to stop the playback and read the parts.

Install

brew install omarish/tap/tokdump
Source →