← Writing

Omar Bohsali

Keeping a Codebase Legible to Models

Ask a model to write a lock-free ring buffer in an empty directory and you will have working code in fifteen seconds. Ask the same model to ship a routine feature inside a three-year-old repository and you will often watch it spend twenty turns reading the wrong files, editing a dead helper, and handing back a pull request that takes a senior engineer an hour to untangle.

tokdumptokdump showing where a string splits into tokens.tokdump$printf 'Hello, world!' | tokdump0000000 13225 11 2375 0 Hello|,|·world|!0000004$printf '💯🔥' | tokdump -n0000000 31446 107 \xf0\x9f\x92|\xaf0000002 96606 🔥0000003$Three columns: offset, token IDs, and thetext they cover. The layout hexdump has usedsince 1978.Two emoji. The first costs two tokens andsplits mid-character; the second is a singletoken.Narrow: two token IDs per row instead offour.
tokdump prints byte offsets in column 1, o200k_base token IDs in column 2, and exact token boundaries in column 3. Hover any blue command token in line 1 to inspect the invocation.

Most teams blame the model or write longer prompt files. Reading our own agent runs in Fractal OS against the commits they produced, we kept finding something simpler: the repository itself had become illegible.

Seeing what the tokenizer sees

People read source files by line and indentation. Models read a flat stream of integer token IDs emitted by a byte-level BPE state machine11. Specifically o200k_base (200,000 vocabulary entries). Notice in the terminal above how the two emoji 💯🔥 (8 UTF-8 bytes) split across three tokens: 31446, 107, and 96606.. Because those boundaries don’t show up in an editor or a GitHub diff, teams routinely commit generated fixtures, inline SVGs, or sprawling lockfiles that quietly eat a large share of an agent’s context window on every single read.

You pay for those tokens twice: once on the inference bill, and a second time when useful architectural context gets pushed out of working memory. We wrote tc as a companion to wc(1) so both humans and CI pipelines can check the token weight of a directory in milliseconds:

tctc counting tokens in three files, then in a pipeline.tc$tc README.md main.go encoding.go2288 README.md1304 main.go389 encoding.go3981 total$printf 'Explain this diff.' | tc4$Counts tokens, not characters or words.Always o200k_base -- the encoding GPT-4oreads.One count per file, then a total. The sameshape wc has printed since 1971.With no files it reads stdin, so it dropsinto a pipeline without ceremony.
tc counts o200k_base tokens across files or standard input with the same flags and output shape as wc(1).

Where the turns go

While building Fractal OS, we record each sub-agent session alongside the commit it produced22. Each <session> node in project.xml logs first-edit-turn, read-calls, edit-calls, and token usage so we can see where an agent spends its time before touching a file.. What we see is not smooth degradation as a codebase grows. In a subsystem with clean module boundaries, an agent reads a handful of files and starts editing. In one that has accumulated duplicate helpers, route drift, and stale notes, the same agent spends most of its turns reading before it touches anything, and the pull request it hands back is the one a senior engineer has to untangle. That is the moment an engineering organization feels like AI tools have made it slower rather than faster.

Checking prose like code

In a human-only workflow, stale documentation is a minor annoyance; a developer asks a teammate in Slack and moves on. In an agent workflow, every markdown file in the repository is treated as ground truth. If two docs disagree on how routes or CSS classes are structured, agents will alternate between both patterns on every commit.

We wrote metadata-enforcer to validate Markdown frontmatter and repository docs against declarative Python schemas in CI, failing the build the moment instructions drift:

metadata-enforcermetadata-enforcer catching a field that drifted.metadata-enforcer$metadata-enforcer check --schema @v.schema
metadata-enforcer check validating frontmatter schemas across a directory of notes.

A shared state for plans and runs

Keeping the repository legible solves half the problem; the other half is keeping track of the plan itself. Chat scrollback is a terrible place to run an engineering project. In Fractal OS, both human leads and autonomous sub-agents read and update a single XML document (project.xml) that ties every task to its verification criteria and execution history:

<task id="srclint-metrics" wbs="1.6" status="in-progress" assigned-agent="omarish-agy">
  <outcome id="oc-legibility" status="met">
    <criterion status="met">Deterministic commit-to-transcript linking</criterion>
  </outcome>
  <session id="run-srclint-d9a891dc" model="gemini-3.1-pro"
           first-edit-turn="15" read-calls="228" edit-calls="157" />
</task>

When the plan, the transcript, and the codebase all stay small and structured enough to inspect directly, a very small team can run dozens of parallel changes without losing the thread.