I opened two terminals at 08:20 and closed them at 19:40.
One terminal held a free server box. The other held an agent writing patches.
By lunch I had three diffs and no idea which run produced which.
This is a letter to that version of me. It is also a workflow you can copy.
It uses free model access as a second reader, and a free server as the test runner.
Both are operator-supplied availability claims, not performance promises.
What the day actually cost
I lost roughly three hours re-running the loop. I could not match diffs to runs.
I lost another two hours debugging a patch a reviewer model had blessed.
The rest went to a fixture directory that changed hash between runs.
None of those failures were the model's fault. All three were missing counters.
Mistake 1: I let the harness count attempts, not me
The agent retried. The wrapper retried. The shell loop retried again.
Three layers of retry, one log file, zero timestamps. That is not observability.
Fix: write a ledger row before the first model call, not after the last.
The ledger is a TSV file. It is boring and it saved the next day.
#!/usr/bin/env bash
# Adapted example. Not run as-is; substitute your own paths and commands.
set -euo pipefail
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
ART="./runs/$RUN_ID"
mkdir -p "$ART"
LEDGER="./ledger.tsv"
log() { printf '%s\t%s\t%s\t%s\n' "$RUN_ID" "$1" "$2" "$3" >> "$LEDGER"; }
FIXTURE_HASH="$(find fixtures -type f -print0 | sort -z \
| xargs -0 sha256sum | sha256sum | cut -d' ' -f1)"
log fixture "$FIXTURE_HASH" "frozen-input"
log attempt "1" "start"
The rule I now follow: no ledger row, no model call.
A run without a row is a rumour, even if the diff looks correct.
Mistake 2: I treated reviewer agreement as a test
I asked a model to check a patch. It said the patch looked correct.
I merged it. A unit test then failed on an off-by-one in the range check.
A reviewer model produces an opinion. Your test suite produces an exit code.
Only one of those two things can block a merge in your CI.
Fix: the reviewer sees the diff and the failing output. Nothing else.
Feed it the smallest context that still explains the defect.
# Adapted example: reviewer input is diff + failing test output only.
git diff --no-color > "$ART/patch.diff"
pytest -q 2>&1 | tail -n 40 > "$ART/test_tail.txt" || true
# Build the reviewer payload from those two files, not from the repo.
printf '%s\n' 'Review this diff against this test failure.' \
'Return JSON: {"verdict":"agree|disagree","reason":"..."}' \
'--- DIFF ---' > "$ART/review_prompt.txt"
cat "$ART/patch.diff" >> "$ART/review_prompt.txt"
printf '%s\n' '--- FAILURE ---' >> "$ART/review_prompt.txt"
cat "$ART/test_tail.txt" >> "$ART/review_prompt.txt"
Then compare two things: the reviewer verdict and the test exit code.
If they disagree, stop. A human reads the run before anything else happens.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Mistake 3: I ran the writer and the reviewer in one workspace
The writer agent wrote scratch files. The reviewer read that polluted directory.
The next run's fixture hash no longer matched the ledger, so the diff was unprovable.
Fix: separate directories, one hash gate, and a write-path allowlist.
Run the gate before any shell command executes. Not after.
# 1) Plan must be small, typed, and declare its write paths.
jq -e '
(.steps | length) > 0 and
(.steps | length) <= 5 and
has("write_paths") and
all(.steps[]; (.cmd | type) == "string")
' plan.json > /dev/null
# 2) Write paths must stay inside the allowed trees.
jq -r '.write_paths[]' plan.json | while read -r p; do
case "$p" in
./src/*|./tests/*) ;;
*) echo "refusing path: $p" >&2; exit 3 ;;
esac
done
# 3) Fixture pack must be unchanged since the ledger row was written.
NOW="$(find fixtures -type f -print0 | sort -z \
| xargs -0 sha256sum | sha256sum | cut -d' ' -f1)"
[ "$NOW" = "$FIXTURE_HASH" ] || { echo "fixture drift" >&2; exit 4; }
The loop that survived the week
- Freeze the fixture pack and log its hash before anything else.
- Write the plan as JSON. Validate it with
jqand the allowlist gate. - Run the test suite on the free server option, capture exit code and tail.
- Send only the diff and the failure tail to the free model access pass.
- Compare verdict with exit code. Disagreement halts the run.
- Append one ledger row per step, including wall-clock seconds.
Six steps produce a receipt you can replay on Friday. That is the whole point.
When free capacity is the wrong tool
Free access is availability, not a throughput guarantee. I plan short bursts, not deadlines.
I also assume nothing about persistence. Artifacts come back to my machine immediately.
Three rows are judgement calls. Two rows are hard limits, not preferences.
Limitations and who should skip this
Free model access and the free server option are operator-supplied claims. Verify terms yourself.
I did not measure throughput, and I would not schedule a release on it.
Skip this workflow if your tests need production credentials or regulated data.
Skip it if you need contractual uptime or data residency guarantees.
Skip it if a human reviewer cannot read the ledger before merge.
What I would tell 08:20-me
Write the counter before you write the prompt. Write the hash before the diff.
When the reviewer and the test disagree, believe the test and call a human.
If you want to try this, run the six-step loop once against free model access and the free server option. Keep the ledger even if you stop using both tomorrow.#ai #testing #devops #productivity #software #coding #development #engineering #inclusive #community
Letter to 08:20-Me: Stop Auditing the Agent, Audit the Counters
