At some point I had a CLAUDE.md with more than a dozen behavioral guidelines. I forced TDD. I installed token-saving skills. None of it felt wrong; all of it felt productive.
Then, without any particular evidence, I started drifting. I dropped process enforcement and put the energy into context engineering and deterministic tooling: linters, test suites, automated checks that run regardless of what the model decides. The results felt better. But I didn’t have numbers; I had vibes.
The experiments below give me numbers. The numbers largely confirm the drift.
Forced TDD Made Things Worse in Both Studies
Birgitta Böckeler (martinfowler.com, Aug 2026) ran “fully agentic TDD” on three greenfield tasks of small, medium, and large scope, with the agent executing its own red-green-refactor loop. She compared TDD and non-TDD runs (two of each per batch) and had Opus evaluate the results blind. Non-TDD solutions took most of the top rankings; no consistent quality advantage for TDD emerged. TDD consumed 3x to 8.5x the tokens.
The conceptual reason is worth sitting with. TDD’s core benefits manage fear, create design pressure, and ensure you see red “for the right reason.” Those are human-centered payoffs. When the same agent writes both the test and the implementation, “red” only proves execution happened. Böckeler observed tautological tests recomputing expected values with the same logic as the implementation: the test confirming itself, not the behavior. Her recommendation shifts from process narration to outcome inspection: mutation testing, static analysis, structural reviews.
The second study, kunchenguid’s ProgramBench (June 2026), ran 192 matched pairs on gpt-5.5. CLI reverse-engineering tasks, graded by hidden test suites. Forced TDD dropped the pass rate from 52.4% to 48.8% (-3.6pp, p<0.0001). Cost went from $1.12 to $1.73 (+55%). Turns went from 40 to 67.
The token anatomy is the interesting part. Output tokens grew only +26%, but input grew +112% and reasoning turns grew +248%. Every red-green micro-cycle re-bills the entire context. kunchenguid’s summary: “The money buys process, not product.”
Without the hidden spec, self-written tests were insufficient, and “minimum code to pass the tests” shipped incomplete products. One TUI task delivered without the terminal interaction loop.
Both authors flag limits clearly: small samples, one-shot hidden-spec settings, nothing resembling a long-lived codebase with a known spec. This doesn’t refute human TDD. But if you’re running forced TDD in an agentic setting and haven’t measured the outcome, you’re paying 3x to 8.5x for a belief.
Behavioral Guidelines Narrowed Scope, Not Code
Same ProgramBench setup. kunchenguid injected four principles via AGENTS.md: “Think Before Coding, Simplicity First, Surgical Changes, Goal-Driven Execution.” Pass rate dropped from 53.7% to 51.5% (-2.2pp, p=0.005). 79 wins, 111 losses, 2 ties. Cost up roughly 5%.
The guidelines didn’t make code simpler; they made scope smaller. “Simplicity First” caused the agent to build limited custom implementations instead of delegating to bundled engines like SQLite. Edge cases got quietly dropped. kunchenguid’s observation: “It reasons itself out of completeness.”
That’s a specific failure mode, not a universal one. A guideline that says “don’t over-engineer” gets interpreted literally, and the model drops whatever it judges as over-engineering. Against a comprehensive hidden test suite, that costs you.
Token-Saving Skills: One Real Signal, Two Scoreboard Problems
Denis Shiryaev at JetBrains measured three popular techniques in a three-part series (July 2026).
Caveman (terse caveman-speak prompting): advertised 65% savings, measured -8.5% output tokens. Technical output (code, diffs, tool calls) can’t be compressed; only narration shrinks. The savings figure itself collapsed on replication: k=1 on 10 tasks showed -29.5%, three repeats brought it to -6.7%. Never trust k=1.
RTK (shell-output compression hook): RTK’s own dashboard claimed “96.2 million tokens saved.” The actual bill went up +7.6% at low effort (p=.004). Built-in Read/Grep bypass the hook; only about 33% of Bash calls were reachable, for a theoretical maximum saving of ~3% of total input. A tool’s self-reported savings are a claim about its counterfactual, not about your bill. At high effort the difference vanished to +0.1%.
Ponytail (minimal-code guidance): the one real signal. Median cost -10.3% (p=.004), code length -15.4% (advertised 54%). Savings concentrate where there’s room to over-build; near zero where code is already minimal. One trap: installing just the skill file produced zero self-activation in 10 sessions; plugin injection was required.
These Measurements Have Expiration Dates
Every study above names its model version (Sonnet 4.6, Opus 4.8, gpt-5.5, claude-sonnet-5), and every author acknowledges these are snapshots. Most agent practices are scaffolding for current model weaknesses. As models improve, yesterday’s best practice becomes today’s tax.
A few data points:
- Ivett Ördög’s hypothesis, quoted by Böckeler: training data contains many requirement -> finished-function mappings but few genuine step-by-step TDD traces. Forcing TDD fights the model’s learned way of working.
- RTK’s effect flipped between effort levels (+7.6% at low, +0.1% at high), suggesting it interacts with model capability, not just context size.
- Anthropic’s context-engineering post (Thariq Shihipar, July 2026): they removed over 80% of Claude Code’s system prompt for Claude 5-generation models (Opus 5, Fable 5) with no measurable coding-eval degradation. The stated principle: “Let Claude use judgement.” Keep CLAUDE.md light.
- Boris Cherny (creator of Claude Code), YC interview, July 2026: “Every six months, delete your CLAUDE.md, delete your skills, delete your hooks.” Re-add instructions only when you observe a concrete failure.
Measure Before You Adopt a Vibe
A few methodological lessons from these studies:
Small samples and single runs betray you. That -29.5% Caveman figure from one run became -6.7% across three repeats. Run at minimum k=3; paired measurements are better.
Don’t trust a tool’s self-report. The 96.2 million tokens “saved” coexisted with a higher actual bill. The scoreboard measures a counterfactual it invents.
Human-centered practices don’t transfer automatically. TDD’s design pressure and genuine red-phase are human payoffs. The agent writing both sides gets none of them.
Run measurements on your own workload. Pass rates on hidden-spec CLI tasks say nothing about your codebase.
Measure before you adopt a vibe.
What I’m Investing in Instead
I’m down to two categories.
Context engineering: getting the model the right context at the right time, not writing longer instructions. A behavioral guideline in CLAUDE.md is a bet that the model interprets it correctly and applies it to the right situation. Structured context is a more reliable bet.
Deterministic tooling: tests, linters, and automated checks that run regardless of model judgment. “Write good tests” in a prompt has an expiration date. An actual test suite and linter survive model upgrades. When the model improves, the suite still runs. When it regresses on a specific task, the suite still catches it.
Those guidelines stacked in your CLAUDE.md. Have you ever tried removing them and re-running?
References
- Birgitta Böckeler, TDD in the agent loop (martinfowler.com, Aug 2026)
- kunchenguid, Does test-driven development help a coding agent? (June 2026)
- kunchenguid, Do “reduce LLM coding mistakes” guidelines help a coding agent? (June 2026)
- Denis Shiryaev, Does Speaking to Agents Like Cavemen Really Save 65% of Tokens? (JetBrains AI blog, July 2026)
- Denis Shiryaev, Does ‘rtk’ reduce Claude Code token usage? (JetBrains AI blog, July 2026)
- Denis Shiryaev, Ponytail Skill for Claude Code (JetBrains AI blog, July 2026)
- Thariq Shihipar, The new rules of context engineering for Claude 5 generation models (Anthropic, July 2026)
- Boris Cherny, Building Claude Code (YC Root Access, July 2026) · interview video: We Cut 80% of Claude Code’s Prompt