KPI Creator token comparison
4 runs, from the Claude session logs. Built 30-Oct-26 10:20. Tokens include cached re-reads, which cost about a tenth of new input.
What stands out
- First run to latest: ClearCompare 487.8M to Base to Build (Live run) 55.7M, down 88.6% (8.8x)
- Biggest Ovy share: ClearCompare, 100% of tokens in the main chat (average context 502K)
- Extra agent starts: Kestrel Medical (8)
Tokens per run, by agent (millions)
OvyGaryKewpieViktorJenniferStakeholderOther
ClearCompareInline: all agents in one chat
Stages 1 to 4. Includes agent improvements, publishing and the simulation in the same chatKestrel MedicalSubagents (before v1.15)
Stages 1 to 4. Includes runs 1 and 2 (stopped interview starts, 11.0M), the Stage 5 walk-through and marking. Continued in a second worktree (counted together). Ends at the wrap commit, 24 Oct 11:58Lumenwire AudioSubagents (v1.16)
Stages 1 to 4. Includes the Stage 5 walk-through, the sealed check and scoring (10/20). Ends before the Agent Lessons Learned edits and wrap, 27 Oct 10:32. Leaves out the two extra dashboard styles Adam asked for (Experimental C newspaper and D tech magazine)Base to Build (Live run)Subagents (v1.21), live mode, Adam as stakeholder
Stages 1 to 4. First live run, on Adam's own Apple Health data, Adam as stakeholder. Ends at the close, before the wrapImprovements
ClearCompare to Kestrel Medical
6 changes · saved 391.4M · added 18.2M (measured)
Includes the changes made for Kestrel runs 1 and 2
- BeforeEvery agent worked inside one main chat, which got re-read on every callAfterStages run as subagents under Ovy. Only a short hand-back reaches the main chat
- BeforeEvery agent ran on the main chat's model (Opus)AfterA model per agent: Opus for Gary and Viktor, Sonnet for Kewpie and Jennifer, Haiku for stakeholders
- BeforeKPI Creator detail sat in CLAUDE.md and loaded into every session and subagentAfterDetail moved to the Status file, which only Ovy reads
- BeforeThe run shared its chat with agent improvements, publishing and the simulationAfter/kpi-creator command: a clean start for each run
- BeforeThe stakeholder interview ran inside the main chatAfterThe stakeholder replies to Gary by message. Gary reports to Ovy only at the end
- BeforeInterview answers could leak back into Ovy's chat (run 2)AfterStrict one-word hand-backs ("Ready." or "Replied.") keep answers out of Ovy's chat
Kestrel Medical to Lumenwire Audio
7 changes · saved 28.4M · added 24.9M (measured)
Process v1.15 and v1.16
- BeforeChange rounds went to a fresh agent each timeAfterChange rounds go back to the same agent (0 extra starts)
- BeforeOvy ran many Bash calls at each gateAfterOne script call per gate (stage_close.py)
- BeforeOutputs placed on a Miro boardAfterA generated Outputs in Order page, no Miro calls
- BeforeEvery agent read the 25K Process fileAfterEach agent reads only its own card, with a set tool list
- BeforeA separate Test Case Assessor agent scored the runAfterOvy runs the sealed check and scores the run
- BeforeKewpie built the calculations onlyAfterKewpie also finds the root cause of each off-track KPI
- BeforeCharts with notes underneathAfterStory cards beside each chart, plus Newspaper and Industry magazine patterns
Lumenwire Audio to Base to Build (Live run)
8 changes · saved 62.9M net (measured total)
Process v1.20 and v1.21. A live run on Adam's own data, so not like for like: no stakeholder agent and no sealed check
- BeforeMany browser calls to check each dashboardAfterOne browser batch per width (dashboard_check.js)
- Before4 dashboard versions built before Adam pickedAfterAdam picks the story and pattern from text first, then Viktor builds Standard plus 1 Experimental
- BeforeOvy read browser previews in his own chatAfterOvy stays text-only, and the sealed check runs in a short subagent
- BeforeSeveral design rounds per gateAfterOne design round per gate
- BeforeSubagents ran past 300K contextAfterPlanned handover at 150K
- BeforeKewpie ran 65 inline Bash callsAfterKewpie works through one script file
- BeforeNo check of how much of the data each source coversAfterCoverage profile at the start of Stage 2
- BeforeNo token budget per stageAfterA 25M budget across Stages 1 to 4, plus Kewpie's quarter-end pull-forward check
Improvements between projects
Where the tokens went between projects (millions)
Carried to the next projectSavedAddedDashed: estimate
Estimates don't include compounding: a smaller Ovy chat makes every later call cheaper. Target for Stages 1 to 4: 25M. Hover or tap a band for its estimate, actual and note.
Show as table
| Project | Improvement | Kind | Estimate (M) | Actual (M) | Note |
|---|---|---|---|---|---|
| Kestrel Medical | Run kept apart from agent fixes, publishing and the simulation | Saved | 300.0 | 308.3 | 157M of agent fixes and 151M of publishing and simulation work were not in the run chat |
| Kestrel Medical | Stages run as subagents | Saved | 80.3 | 83.1 | The four stage agents' first passes fell from 107.1M to 24.0M. Estimate: 75%, from Gary's inline cost |
| Kestrel Medical | KPI Creator detail moved out of CLAUDE.md | Saved | 0.4 | Not measured on its own | About 600 tokens on every call. Too small to measure on its own |
| Kestrel Medical | Meeting Script inside the Stage 1 Review | Saved | 0.5 | Not measured on its own | No separate HTML file. Too small to measure on its own |
| Kestrel Medical | Blind stakeholder agent, interviewed by message | Added | 4.0 | 0.8 | Sharon on Haiku. Ovy's wake-ups during the interview sit inside Ovy's line |
| Kestrel Medical | Test Case Assessor scoring | Added | 1.0 | 1.0 | Scores the finished run against the sealed marking note |
| Kestrel Medical | Fresh agents for change rounds | Added | Not estimated | 16.4 | Not estimated. 6 extra agent starts for fixes. v1.15 sends these to the same agent |
| Lumenwire Audio | Interview restarts don't recur | Saved | 11.0 | 11.0 | No restarts: Gary started once and the interview ran through |
| Lumenwire Audio | /kpi-creator skips /prime (no saving: /prime wasn't running) | Saved | 0.0 | 0.0 | The 11.5M estimate was wrong. /prime didn't run in Kestrel either: both runs started at about 80K, the app's fixed load (tools, system prompt, CLAUDE.md, listings) |
| Lumenwire Audio | Change rounds go back to the same agent | Saved | 9.0 | 16.4 | No extra agent starts (Kestrel had 8, 16.4M). The change rounds now sit inside each agent's own line |
| Lumenwire Audio | One script call per gate (stage_close.py) | Saved | 4.0 | Not measured on its own | Replaces many Bash calls in Ovy's chat |
| Lumenwire Audio | Previews read as text, at most one half-scale screenshot | Saved | 4.0 | Not measured on its own | No screenshot loops |
| Lumenwire Audio | No Miro: generated Outputs in Order page | Saved | 3.0 | Not measured on its own | No Miro tool calls or results in Ovy's chat |
| Lumenwire Audio | Each agent reads only its card, with a set tool list | Saved | 2.5 | Not measured on its own | 7K to 10K card instead of the 25K Process file |
| Lumenwire Audio | Quiet interview wake-ups | Saved | 1.5 | Not measured on its own | No tool calls on progress hand-backs |
| Lumenwire Audio | No Interview Notes, 3-line gate summary, short hand-backs | Saved | 1.0 | Not measured on its own | Less written and re-read |
| Lumenwire Audio | Token count logged at each approval | Added | 0.3 | Not measured on its own | One token_report.py call per stage |
| Lumenwire Audio | Test Case Assessor retired | Saved | 1.0 | 1.0 | No separate marking agent ran |
| Lumenwire Audio | Sealed check and scoring in Ovy's chat (v1.16) | Added | 3.0 | 3.6 | Ovy's calls from the go-live approval to the end of the sealed check, including one context summary |
| Lumenwire Audio | Kewpie's root cause analysis (v1.16) | Added | 2.5 | Not measured on its own | Clues, suspects and tests on the data, written up as Root Cause Findings. About 15 more Kewpie calls |
| Lumenwire Audio | Viktor's story cards and Newspaper pattern (v1.16) | Added | 1.5 | Not measured on its own | Longer guides and a larger Chart Standards Samples page to read, plus a side-summary card for every chart |
| Lumenwire Audio | Head of Data feedback at every gate (v1.16) | Added | 1.5 | Not measured on its own | Ovy logs Keep, Change and Decision at each gate, plus the final and go-live approvals |
| Lumenwire Audio | Longer agent cards (v1.16) | Added | 0.5 | Not measured on its own | Every card gains The Human section. Ovy's card grows by about 1.2K tokens, re-read on each call |
| Lumenwire Audio | Design and Stage 5 change rounds in Viktor | Added | Not estimated | 21.3 | Not estimated. Declutter and whitespace, front-page spacing, Why spacing (15.5M), then the Stage 5 quick fixes (5.9M). The font change to Experimental D is left out with the extra styles |
| Base to Build (Live run) | Dashboard checks in one call per width (dashboard_check.js) | Saved | 17.5 | Not measured on its own | Viktor and Jennifer made 119 javascript checks, 85 resizes and 61 screenshots on Lumenwire, each re-reading a 150K to 250K chat. Now one browser batch per width, one desktop and one phone screenshot per version |
| Base to Build (Live run) | Story and chart choices picked before any HTML, Standard plus 1 Experimental | Saved | 15.0 | Not measured on its own | Adam picks the pattern from Viktor's text story first, so only 2 versions are built and checked, not 4 |
| Base to Build (Live run) | Ovy stays text-only; sealed check in a short subagent | Saved | 12.5 | Not measured on its own | Ovy's chat reached about 300K. No browser previews read in his chat, and the sealed check runs in a fresh subagent (the coverage check subagent cost 1.0M) |
| Base to Build (Live run) | One design round per gate | Saved | 10.0 | Not measured on its own | Adam's Keep / Change list goes to Viktor in one message; polish after Stage 3 approval goes to Jennifer. Lumenwire's design rounds cost 21.3M |
| Base to Build (Live run) | Planned handover at 150K context | Saved | 7.5 | Not measured on its own | A subagent past 150K saves a state note and a fresh copy carries on from it. Kewpie reached 334K and Viktor 365K. Tested on Viktor first |
| Base to Build (Live run) | Kewpie works through one script file | Saved | 6.5 | Not measured on its own | One calc script re-run, instead of 65 inline Bash calls that each stay in her chat |
| Base to Build (Live run) | Coverage profile in describe_data.py | Added | 0.5 | Not measured on its own | One extra table at the start of Stage 2. It flags partial sources, such as telemetry seeing 73% of invoicing OEMs on Lumenwire |
| Saltbush Homes | Stakeholder agent, interviewed by message | Added | 0.6 | Not run yet | Lumenwire's measured Haiku stakeholder. The Live run had none: Adam answered live |
| Saltbush Homes | Sealed check in a short subagent | Added | 1.0 | Not run yet | What the coverage check subagent cost on Lumenwire. The Live run had no sealed check |
Run summary
| Run | ClearCompare | Kestrel Medical | Lumenwire Audio | Base to Build (Live run) |
|---|---|---|---|---|
| Started | 20-Oct-26 11:23 | 23-Oct-26 09:14 | 26-Oct-26 19:36 | 29-Oct-26 20:10 |
| Setup | Inline: all agents in one chat | Subagents (before v1.15) | Subagents (v1.16) | Subagents (v1.21), live mode, Adam as stakeholder |
| Stages | 4 | 4 | 4 | 4 |
| Total (M) | 487.8 | 124.4 | 118.6 | 55.7 |
| Per stage (M) | 122.0 | 31.1 | 29.7 | 13.9 |
| Calls | 969 | 738 | 684 | 390 |
| Output (M) | 0.93 | 0.65 | 0.70 | 0.41 |
| Ovy share | 100% | 63% | 37% | 37% |
| Ovy average context | 502K | 246K | 209K | 153K |
| Extra agent starts | 0 | 8 | 0 | 0 |
| Run | Started | Setup | Stages | Total (M) | Per stage (M) | Calls | Output (M) | Ovy share | Ovy average context | Extra agent starts |
|---|---|---|---|---|---|---|---|---|---|---|
| ClearCompare | 20-Oct-26 11:23 | Inline: all agents in one chat | 4 | 487.8 | 122.0 | 969 | 0.93 | 100% | 502K | 0 |
| Kestrel Medical | 23-Oct-26 09:14 | Subagents (before v1.15) | 4 | 124.4 | 31.1 | 738 | 0.65 | 63% | 246K | 8 |
| Lumenwire Audio | 26-Oct-26 19:36 | Subagents (v1.16) | 4 | 118.6 | 29.7 | 684 | 0.70 | 37% | 209K | 0 |
| Base to Build (Live run) | 29-Oct-26 20:10 | Subagents (v1.21), live mode, Adam as stakeholder | 4 | 55.7 | 13.9 | 390 | 0.41 | 37% | 153K | 0 |
By agent (millions)
One dot is 1M tokens. Dots are the same size in every run, so a bigger circle means a bigger run.
Each run's tokens by agent (millions)
OvyGaryKewpieViktorJenniferStakeholderOther
Each bar is one run, split by agent. Bar height is the agent's share of that run; the numbers are millions of tokens. Ribbons follow each agent from run to run. Values too small to label sit under their bar.