GobstopperBenchmarks
Benchmarking compaction is split between offline structural checks and live provider trials. We do not claim subscription savings without completed live evidence.
Offline proxy benchmark
cargo run --example bench_strategies --release generates synthetic Codex transcripts with varying tool-output history and prints a CSV of actual byte reduction, projected token reduction, elapsed time, and structural integrity per strategy.
Live qualification
Live trials resume a real session after compaction and measure:
- Task completion on a held-out second prompt.
- Observed input and output tokens from the provider.
- Latency and re-fetch count.
- Verbatim recall of elided tool output where relevant.
Live API token comparison
The same 333k-token Claude session was restored from the gobstopper vault and asked the same resume question under four conditions. The question was: "What were we working on? Briefly state the current task and the most recent concrete decision or conclusion, if any." Token numbers are the observed provider usage for the resume turn (input = cache read + cache creation + uncached input; output = response tokens).
| condition | input tokens | output tokens | recalled the standing task? |
|---|---|---|---|
| none (original) | 312,722 | 1,405 | yes — npm unification and stalled renames |
gobstopper elide | 219,167 | 1,052 | yes — same standing task, stalled renames |
gobstopper compacted | 220,447 | 621 | yes — same standing task from the state-card digest |
Claude --autocompact 100 | 56,300 | 416 | no — incorrectly claimed the renames were already done and published |
gobstopper elide and compacted both cut the resume context by about 30% while keeping the answer accurate. Claude's native --autocompact 100 cut the resume context by ~82% but produced a confident, inaccurate summary of the session.
Codex live comparison
The same resume question was asked on a 101k-token Codex session (a real BEAM-benchmark thread) under three conditions. Codex does not expose a one-shot native compact to test against.
| condition | input tokens | output tokens | recalled the standing task? |
|---|---|---|---|
| none (original) | 101,275 | 244 | yes — Oh's memory benchmark and the 0.60 expansion gate |
gobstopper elide | 57,980 | 83 | yes — same 0.545 score and 0.60 gate |
gobstopper compacted | 34,503 | 159 | yes — same BEAM experiment and expansion gate |
On Codex, compacted cut resume input tokens by 66% andelide cut them by 43%, both with accurate answers. Every pre- and post-state is in the vault, so you can gobstopper diffthe exact structural changes and choose the strategy that matches your tolerance for recall loss.
Prefix preservation and provider cache
The cache_aware strategy elides the latest stale tool outputs before the protected tail instead of the oldest, so the conversation prefix stays byte-identical. On a 339k-token Claude session at a 310k floor, the same resume question was asked under three conditions (same session, restored between runs viagobstopper undo --in-place) and the provider's real cache counters were read from the API response:
| condition | cache read | cache creation | file prefix preserved | cost | accurate? |
|---|---|---|---|---|---|
| none (original) | 10,010 | 325,647 | — | $6.53 | yes |
gobstopper cache_aware | 13,536 | 258,517 | 107,884 tokens | $5.19 | yes |
gobstopper compacted | 13,536 | 257,505 | 6,639 tokens | $5.18 | yes |
All three answers were accurate. Both strategies cut cache-write tokens by ~21% (~20% lower cost on the resume turn). The honest caveat: Claude Code's cache breakpoints boundcache_read at ~13.5k in every condition, so the extra prefix preservation does not translate into more provider cache hits today — it pays off in smaller gobstopper diffaudits and cheaper record dedup across repeated compactions.
Results
| intervention | records | removed | added | resumed? |
|---|---|---|---|---|
| original session | 5,244 | — | — | yes |
Claude --autocompact 100 | 5,303 | 0 | 59 | yes |
gobstopper compacted | 5,247 | 43 | 46 (43 stubs + 3 digest) | yes |
On the same 333k-token real Claude session, native --autocompact 100appended 59 records and removed none. gobstopper's in-placecompacted strategy removed 43 stale tool records, injected a resumable state-card digest, and claude --resume succeeded with the model recalling the last user prompt and current task state. The structural diff is available in the vault viagobstopper diff.