Imagine the same prompt, the same repository, and the same command, run on two different afternoons. On the first afternoon the agent finishes quickly and the test suite is green; on the second it edits four extra files and leaves two tests red. Nothing in the repository changed between those runs, and that is precisely why the second outcome is difficult to defend in a code review.
Most comparisons of free AI coding tiers begin and end with price. The question that actually changes your workflow is whether a tier behaves consistently enough that you can build a habit around it, because a cheap tool with a wide outcome spread quietly spends your review time instead. This article converts that spread into a small ledger you can run in one sitting, and it retires four claims that developers repeat whenever tiers are compared.
The format is a myth-busting FAQ, because myths are what get repeated in chat threads while the measurements stay private. Each claim below comes with a cheap check and a corrected mental model, and none of the checks requires a paid plan to be interesting.
Myth 1: The same prompt produces the same run
Repeat a task eight times against the same commit and you will rarely get eight identical diffs. Sampling settings, tool-call ordering, network retries, and the order in which files are read all push the run in slightly different directions. A single green run is therefore an anecdote about one sample, not a property of the tier.
The corrected mental model is that you are not buying a function, you are sampling a distribution. That sounds abstract until you write the distribution down, and writing it down takes less effort than arguing about it in a pull request. Once you have rows, the conversation moves from "it worked for me" to "median acceptable, tail unacceptable".
Myth 2: A green run verifies the workflow
A passing test suite proves that one sample passed, and it says nothing about how often the next sample will pass. The failure mode is subtle: the agent's diff and the test outcome are recorded together in your head, but only the happy version survives into the team's memory. Recording both columns for every run is the entire fix.
The corrected model separates process from task. Exit code zero tells you the command finished; a test result tells you the change was acceptable; a diff count tells you how much review surface the change created. Those three signals disagree often enough that collapsing them into one boolean will mislead you.
Myth 3: The median is the number that matters
Medians are comfortable and tails are expensive, and free tiers tend to produce tails. If seven runs finish in a few minutes and the eighth takes an hour, your median looks healthy while your afternoon is gone. Anyone who has waited on a stuck job knows the average is not the number that ruins the day.
The corrected model is a tail ratio: worst run divided by median run, computed from the same ledger rows. A tier with a stable median and a large tail ratio is fine for exploration and hostile for anything with a deadline. Compute both numbers before you decide which kind of tool you are holding.
The ledger: eight runs, four columns
The harness below is deliberately boring, and you should adapt the one line that invokes your agent. It records the commit you started from, resets the worktree between runs, and appends one CSV row per attempt. Treat it as a proposal to run on your own task rather than as a benchmark result, since the numbers you get will depend on your repository.
#!/usr/bin/env bash
# variance_ledger.sh — repeat one task N times, record one row per run
set -uo pipefail
N="${N:-8}"
TASK="${TASK:-tasks/fix_import_order.md}"
OUT="ledger/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"
git rev-parse HEAD > "$OUT/base_commit.txt"
git status --porcelain > "$OUT/base_dirty.txt" # should be empty
for i in $(seq 1 "$N"); do
start=$(date +%s)
# Replace this line with your agent's non-interactive entry point.
agent run --task "$TASK" --no-interactive > "$OUT/run_$i.log" 2>&1
code=$?
end=$(date +%s)
added_deleted=$(git diff --numstat | awk '{a+=$1; d+=$2} END {print (a+0) " " (d+0)}')
printf '%s,%s,%s,%s\n' "$i" "$((end-start))" "$code" "$added_deleted" >> "$OUT/ledger.csv"
git checkout -- . >/dev/null 2>&1
git clean -fdq
done
The reset step is what makes the rows comparable, and skipping it is the most common way to poison a ledger. Running without a clean worktree means run two starts from run one's leftovers, and the whole table then measures compounding edits instead of tier behaviour. Check base_dirty.txt before you trust anything downstream.
Summarising takes a short awk script, because pulling in a dataframe library for eight rows is unnecessary ceremony.
awk -F, '{wall[NR]=$2; n=NR}
END{
for(i=1;i<=n;i++) for(j=i+1;j<=n;j++) if(wall[j]<wall[i]){t=wall[i];wall[i]=wall[j];wall[j]=t}
med=(n%2)?wall[(n+1)/2]:(wall[n/2]+wall[n/2+1])/2
printf "runs=%d min=%ss median=%ss max=%ss tail_ratio=%.2f\n", n, wall[1], med, wall[n], wall[n]/med
}' ledger/*/ledger.csv
Add the task verdict yourself, because wall time cannot tell you whether a diff was correct. A second column holding the exit status of your test command is usually enough, and it keeps the ledger honest when a fast run produces a bad patch.
Myth 4: Variance is a property of the model
Variance usually arrives from the environment wrapped around the model. A shared shell with a warm package cache, a locale that formats numbers differently, and a proxy that silently retries a request are three common sources that have nothing to do with the weights. If you change the environment between runs, you have measured the environment.
The corrected model treats the agent as one variable among several, and it forces you to hold the others fixed or record them. A warm cache and a different locale are environment variables in the literal sense, and both explain why two teams can disagree about the same tier while both are reporting honestly.
Where free access and a free server actually help
Repetition is the step everyone skips, because repetition is what costs money and occupies a machine. MonkeyCode's free model access and free server option change the arithmetic of that step, since an eight-run ledger stops being an expense and becomes an ordinary afternoon task. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The honest version of that claim is narrower than a slogan. Free access makes sampling practical at small n, and a free server keeps the loop off your laptop while you review the previous run; neither makes a tier deterministic. Check the project's own documentation for current allowances and machine characteristics, because quotas and hardware are exactly the details that go stale fastest.
Limitations worth stating plainly: eight runs is a small sample, diff size is not correctness, wall time includes network noise you cannot control, and every repetition consumes allowance you may want later. The ledger also assumes a non-interactive entry point, which not every agent exposes cleanly. Treat the output as a screen for whether a tier deserves careful measurement, not as a benchmark.
Who should not use this approach
Teams that need deterministic output, contractual latency, or regulated audit trails will not be served by sampling a free tier at all. Anyone whose task takes an hour per run should shrink the task before running any ledger. And if review capacity is the real bottleneck, variance reduction matters more than cost reduction, so pay for stability rather than collecting cheap rows.
The myth-busting habit generalises beyond any single product: write the claim down, find the cheapest observation that could refute it, and keep the rows. Eight runs and four columns will not settle a platform debate, but they will stop you from repeating a one-sample story as if it were a property. If you want to try the ledger, the free tier is enough to learn whether your own task is stable before you commit real budget to it.#ai #testing #devtools #productivity #software #coding #development #engineering #inclusive #community
Zero Cost Is Not Zero Variance: Four Myths About Repeating an AI Coding Run
