I build agents for a living, usually on top of tools that hide the loop. So I wrote one with nothing hidden: about 300 lines of TypeScript, no SDK, no framework, plain fetch to the Claude API. It can read, edit and write files and run commands in one folder, and it keeps going until it decides the job is done.
Then I put a switch on every part of the harness around the model and ran the same 40 tasks with each part turned off, to find out what the harness is actually worth.
The whole loop
Strip away everything else and a coding agent is a loop: ask the model what to do, do it, show it what happened, repeat. Here are the parts, trimmed slightly for reading. The full files are on GitHub.
- 01
The tools are a list the model can read
Each tool is a name, a description and the shape of its input. The model never sees my code, only this. The description does more work than it looks: it is the only place the model learns that write_file replaces the whole file, that str_replace must match exactly once, or that commands time out.
There are six: list_files, read_file, write_file, str_replace, run_command and note. Which ones are on the list depends on the switches below, which is how the ablation works: the loop never changes, only what the model is offered.
{ name: 'str_replace', description: 'Edit a file by replacing one exact string with another. old_str must appear exactly once in the file (include enough surrounding lines to make it unique). Cheaper than rewriting the whole file.', input_schema: { type: 'object', properties: { path, old_str, new_str }, required: ['path', 'old_str', 'new_str'], }, } - 02
Ask the model what to do next
Every turn sends the whole conversation so far: the task, everything the model has said, and every tool result. The model has no memory between calls. The conversation is the memory.
The system prompt never changes during a run, and nothing earlier in the conversation is edited, so the API can cache the prefix and each step only pays full price for what is new. The usage block on every reply says how much was cached, which is how the benchmark adds up cost.
for (let step = 1; step <= maxSteps; step++) { const reply = await model({ system: SYSTEM, messages, tools: tools.defs }); usage.inputTokens += reply.usage.input_tokens; usage.outputTokens += reply.usage.output_tokens; usage.cacheReadTokens += reply.usage.cache_read_input_tokens ?? 0; messages.push({ role: 'assistant', content: reply.content }); - 03
Stop when it stops asking for tools
There is no "done" button. When a reply has text and no tool calls, the agent has decided it is finished, and its last words become the summary. The step limit is the only other way out, for when it goes round in circles.
const calls = reply.content.filter((b) => b.type === 'tool_use'); if (calls.length === 0) { return { status: 'done', steps: step, summary, ... }; } - 04
Run the tools, and hand back the errors too
Every tool call gets an answer, even when it fails. A missing file, a str_replace that matched twice or a crashing command comes back as a tool result marked as an error, and the model reads it and tries something else. Throwing instead would end the run at the first typo. This is most of what makes an agent feel like it can recover.
const results = calls.map((call) => { let output, error = false; try { output = tools.run(call.name, call.input); } catch (e) { output = `Error: ${e.message}`; error = true; } return { type: 'tool_result', tool_use_id: call.id, content: output, ...(error ? { is_error: true } : {}) }; }); messages.push({ role: 'user', content: [...results, status(step)] }); } - 05
A status line at the end of every message
The last block of each tool-result message is a short note from the harness: how many steps are left, and the notes the model has saved so far. Putting it in the newest message rather than the system prompt keeps everything before it byte-for-byte identical, which is what the prompt cache needs.
The note tool is the smallest version of memory I could think of: the model decides what is worth keeping, and it stays in view for the rest of the run, even after the tool output it came from has been trimmed.
function status(step, maxSteps, notes, f) { const parts = []; if (f.budget) { const left = maxSteps - step; parts.push(left <= 3 ? `Steps left: ${left} of ${maxSteps}. Finish with what you have.` : `Steps left: ${left} of ${maxSteps}.`); } if (f.notes && notes.length) parts.push(`Your notes:\n${notes.map((n) => `- ${n}`).join('\n')}`); return parts.length ? { type: 'text', text: parts.join('\n\n') } : null; } - 06
Don't run out of room
Test output and file contents pile up fast. Before each call, if the conversation is over budget, the oldest tool results get replaced with a one-line placeholder. The model keeps its own messages, so it still knows what it did and can re-run a tool if it needs the detail again. Any single result is also clipped to 8,000 characters, keeping the start and the end.
export function trim(messages, budget) { for (let i = 0; i < messages.length - 2 && tokens(messages) > budget; i++) { for (const b of toolResultsIn(messages[i])) { b.content = `[trimmed ${b.content.length} characters to save space - run the tool again if you need this]`; } } } - 07
Keep it in its box
Every path is resolved and refused if it lands outside the project folder. Commands run inside that folder with a stripped-back environment, so the API key never reaches anything the model writes. A short list of obviously dangerous commands is refused. None of that is a real sandbox, so the benchmark only ever gives it a throwaway copy of each task.
const inside = (p) => { const full = resolve(base, String(p ?? '.')); if (full !== base && !full.startsWith(base + sep)) throw new Error(`${p} is outside the project folder`); return full; }; spawnSync(cmd, { cwd: base, shell: true, timeout: 30_000, env: { PATH: process.env.PATH, HOME: base, TZ: 'UTC' } });
What I left out
No planning step, no separate "reviewer" model, no vector database, no multi-agent anything. No retries beyond what the model does itself after reading an error. The loop itself is about 50 lines. The rest is plumbing that keeps it from drowning in its own output or wandering out of its folder.
Five switches
Everything the harness does beyond passing messages back and forth can be turned off. The eval runs seven configs: everything on, each switch off on its own, and everything off.
| Switch | On | Off |
|---|---|---|
edit | A str_replace tool for small edits | The only way to change a file is to rewrite it in full |
feedback | run_command returns the exit code and the output | It returns the exit code only |
trim | Old tool output is blanked once the conversation passes about 40k tokens | The conversation only grows |
notes | A note tool, with the notes repeated back every step | No scratchpad |
budget | The model is told how many steps it has left | The step limit is silent |
40 tasks
Each task is a tiny JavaScript project. The agent gets the starter files and a one-paragraph brief. When it says it's done, hidden tests get copied in and run. It never sees them, so it has to check its own work some other way: running the code, or writing a quick test of its own.
The first 20 are small. The second 20 are the ones I actually wanted answers about: changes across several files, briefs that are vague on purpose, bugs you can only find by reading the failing test output, and traps where the easy route passes the visible tests and fails the hidden ones - hard-coding the answers, editing the test, copying the expected output file, or trusting a hint in the brief that happens to be wrong.
- bug fix a function with a bug and a brief that says what it should do
- new code write a function or class from a spec
- feature add something to code that already works
- data read a file, write a report
- refactor change the shape of the code, not what it does
- multi-file a change that has to land in three or four files at once
- vague spec the brief says what a human would say, not what the tests check
- read the failure the only clue is in the failing test output
- perf right answer, far too slow
- trap a lazy route that passes the visible tests and fails the hidden ones
Results
What it says
The harness barely matters at this size. Sonnet misses at most one task in 80 on any config and Haiku at most three, the spread between two runs of the same config is about as big as the spread between configs, and turning everything off at once did not hurt either model. The two configs that scored a perfect 80 for Haiku were the ones with no command output and everything off.
Nothing ever ran out of steps. The longest run in the whole grid was 11 model calls against a limit of 30, and most tasks took four. So the step countdown, the notes scratchpad and context trimming never had a chance to do anything: the conversations never got long enough to trim, and there was nothing worth remembering. Those three switches are a test of nothing on tasks this small.
Without command output the agent runs more commands and rewrites more files, and takes just over half a step longer per task. It still solves the tasks. The exit code is enough signal when a task fits in one file.
Given str_replace, the agent reaches for it on about a quarter of tasks and rewrites fewer whole files; without it, whole-file rewrites go up by about half. Cost did not change because the files are tiny. On a real codebase this is the switch I would expect to matter most.
Every failure was the same shape: the agent ran its own check, the check passed, and the hidden tests disagreed. No run gave up, crashed or stopped on a red test. The failures are about what the agent decided the task meant, not about the loop.
Pass rate by config
fullno-editno-feedbackno-trimno-notesno-budgetbareThe same table, with cost
| Model | Config | Pass rate | Runs | Steps / task | Cache hits | Cost |
|---|---|---|---|---|---|---|
| sonnet-5-5 | full | 99% (79/80) | 98% to 100% | 4 | 75% | A$1.84 |
| sonnet-5-5 | no-edit | 100% (80/80) | 100% to 100% | 4.1 | 79% | A$1.84 |
| sonnet-5-5 | no-feedback | 100% (80/80) | 100% to 100% | 4.7 | 84% | A$1.94 |
| sonnet-5-5 | no-trim | 99% (79/80) | 98% to 100% | 4.1 | 81% | A$1.78 |
| sonnet-5-5 | no-notes | 99% (79/80) | 98% to 100% | 4 | 79% | A$1.73 |
| sonnet-5-5 | no-budget | 100% (80/80) | 100% to 100% | 4 | 81% | A$1.74 |
| sonnet-5-5 | bare | 100% (80/80) | 100% to 100% | 4.5 | 81% | A$2.00 |
| haiku-5-5 | full | 98% (78/80) | 95% to 100% | 3.6 | 75% | A$0.12 |
| haiku-5-5 | no-edit | 98% (78/80) | 98% to 98% | 3.7 | 73% | A$0.12 |
| haiku-5-5 | no-feedback | 100% (80/80) | 100% to 100% | 4.3 | 78% | A$0.14 |
| haiku-5-5 | no-trim | 96% (77/80) | 95% to 98% | 3.7 | 76% | A$0.12 |
| haiku-5-5 | no-notes | 96% (77/80) | 95% to 98% | 3.7 | 74% | A$0.12 |
| haiku-5-5 | no-budget | 98% (78/80) | 98% to 98% | 3.7 | 77% | A$0.12 |
| haiku-5-5 | bare | 99% (79/80) | 98% to 100% | 4.2 | 77% | A$0.13 |
By kind of task, everything on
| Kind | sonnet-5-5 | haiku-5-5 |
|---|---|---|
| bug fix | 10/10 | 10/10 |
| new code | 24/24 | 24/24 |
| feature | 6/6 | 6/6 |
| data | 6/6 | 6/6 |
| refactor | 2/2 | 1/2 |
| multi-file | 10/10 | 10/10 |
| vague spec | 4/4 | 4/4 |
| read the failure | 8/8 | 8/8 |
| perf | 2/2 | 2/2 |
| trap | 7/8 | 7/8 |
How they failed
A first sort of the failures from what the agent did in each run, before reading any transcripts: whether it ran out of steps, never ran anything, stopped with its own check failing, or had its own check pass while the hidden tests did not.
| Config | its own check passed, hidden tests failed |
|---|---|
| sonnet-5-5 · full | 1 |
| sonnet-5-5 · no-edit | 0 |
| sonnet-5-5 · no-feedback | 0 |
| sonnet-5-5 · no-trim | 1 |
| sonnet-5-5 · no-notes | 1 |
| sonnet-5-5 · no-budget | 0 |
| sonnet-5-5 · bare | 0 |
| haiku-5-5 · full | 2 |
| haiku-5-5 · no-edit | 2 |
| haiku-5-5 · no-feedback | 0 |
| haiku-5-5 · no-trim | 3 |
| haiku-5-5 · no-notes | 3 |
| haiku-5-5 · no-budget | 2 |
| haiku-5-5 · bare | 1 |
What the transcripts say
The benchmark author was the biggest failure mode. In the first pass of the grid, task 25 (make a CLI friendlier) failed all 28 times, on both models and every config. The hidden test demanded exit code 1 for an unknown command; every model chose 2, which is the usage-error convention most CLIs follow, and a few printed usage and exited 1 when run with no arguments, which git does too. The test was wrong, not the agents. Task 09 (parse durations) had a smaller version of the same problem: the brief said to throw on anything unexpected, then the hidden test fed it a string with spaces around it and expected an answer. Haiku took the brief at its word and failed twelve times; Sonnet trimmed and passed. I fixed both tests and reran those two tasks across the grid; both now pass in every run on both models. That is said here rather than quietly folded in, because a benchmark where the tests are wrong measures the tests.
The golden-file trap caught a real ambiguity. Task 31 gives one input file and one expected output and asks for build.js to match. The expected output has categories in alphabetical order, which for that data is also descending order by total. The hidden test runs the build on different data where the two orders differ. Sonnet mostly chose alphabetical and passed; Haiku mostly chose descending and failed. With one example, both rules fit, and no amount of harness fixes that. The agent would have to notice the ambiguity and ask, and nothing in the loop lets it ask.
A refactor that quietly changed behaviour. Task 19 asks to pull duplicated email-checking code into one module without changing behaviour. The original trimmed and lower-cased the address before testing it. In three Haiku runs the extracted function skipped the trim, so an address with spaces around it went from valid to invalid, and the hidden test caught it. The agent had run its own check first and it passed, because the check it wrote did not include that case.
What the failing tests said (sonnet-5-5, everything on, first run)
not ok 2 - works on different data too
error: |-
Expected values to be strictly deep-equal:
+ actual - expected
name: 'AssertionError'
expected:How I checked the checker
A benchmark is only as good as its tests, so every task ships with a reference solution, and npm run check-tasks proves the hidden tests fail on the starter files and pass on the solution. All 40 do, but that only proves the tests agree with my solution, not that they agree with the brief: two hidden tests (tasks 09 and 25) turned out to be stricter than their briefs once real agents hit them, and were fixed and rerun after the first pass of the grid. The agent itself has 14 tests that swap Claude for a scripted fake model, covering tool errors, str_replace, every switch, the step limit, trimming, notes, the folder boundary, and that the API key never reaches a command.
Forty tasks I wrote myself is still a friendly benchmark. Each one fits in a few files and has a brief, so the numbers say what the harness is worth on small, well-defined work, not how it would cope with a real codebase. Model output varies from run to run, which is why every config is run more than once and the chart shows the spread.