#usrbinenv — Public Fediverse posts

A shared free AI bench rarely dies from an empty allowance. It dies from an unowned next step.

The developer conversation this week mixed two moods. There is fatigue with AI tooling, and there is a sprint to ship open-source work. Both push teams toward the same move: pool a free tier, share it, and hope nobody hogs it. Hope is not a handoff.

MonkeyCode is an open-source coding agent project. The operator states that it offers free model access and a free server option. Treat both as availability, not as a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The failure this SOP prevents

A free bench has no queue and no billing alarm. Two people start long runs, a third retries a failure, and the window closes mid-task. Nobody can say which run burned the last of it.

The fix is small and boring. Name three roles, write one card per run, and keep a local token ledger you control. The ledger is yours, so it survives quota changes and product changes.

Three roles, one handoff

The Bench Steward owns the budget line and the ledger file. This person approves a run before it starts and closes it after. The Steward does not write code during a shift.

The Runner owns a single run card at a time. One card equals one goal, one model, one output artifact. A second goal needs a second card, even when the first is nearly done.

The Reviewer owns the accept or reject call. This person reads the diff and the card, never the chat history. If the card cannot explain the change, the change is not ready.

Handoffs move in one direction. Runner hands a filled card to Reviewer. Reviewer hands a verdict to Steward. Steward writes the next slot. No role skips forward, and no role edits another role's section.

Artifact one: the token ledger

Put this file at bench/ledger.sh and keep the data in plain TSV. It reads the usage block from a saved response, so it never guesses.

#!/usr/bin/env bash
# bench/ledger.sh — count tokens per run card, refuse to start over budget
set -euo pipefail

BENCH_DIR="${BENCH_DIR:-bench}"
LEDGER="$BENCH_DIR/token-ledger.tsv"
BUDGET_FILE="$BENCH_DIR/budget.txt"

[[ -f "$BUDGET_FILE" ]] || { echo "missing $BUDGET_FILE" >&2; exit 2; }
BUDGET=$(tr -dc '0-9' < "$BUDGET_FILE")
[[ -n "$BUDGET" ]] || { echo "budget file has no number" >&2; exit 2; }

[[ -f "$LEDGER" ]] || printf 'run_id\towner\tmodel\ttokens\n' > "$LEDGER"

spent() { awk -F'\t' 'NR>1 {s+=$4} END {print s+0}' "$LEDGER"; }

check() {
local now left
now=$(spent)
left=$(( BUDGET - now ))
printf 'budget=%s spent=%s left=%s\n' "$BUDGET" "$now" "$left"
(( left > 0 )) || { echo 'FENCE: no headroom left, stop' >&2; return 1; }
}

record() {
local run_id="$1" owner="$2" model="$3" payload="$4" tokens
tokens=$(jq -r '.usage.total_tokens // empty' "$payload")
[[ -n "$tokens" ]] || { echo "no usage block in $payload" >&2; return 1; }
printf '%s\t%s\t%s\t%s\n' "$run_id" "$owner" "$model" "$tokens" >> "$LEDGER"
echo "recorded $tokens tokens for $run_id"
}

case "${1:-}" in
check) check ;;
record) shift; record "$@" ;;
*) echo 'usage: ledger.sh check | record <run_id> <owner> <model> <response.json>' >&2; exit 64 ;;
esac

The run loop is three commands. Check headroom, do the work, then record.

mkdir -p bench
cat > bench/budget.txt <<'EOF'
400000
EOF

./bench/ledger.sh check
# ... run the agent, save the raw response ...
./bench/ledger.sh record r-2026-10-18-a morgan swe-review response.json

The number in budget.txt is your own planning line, not the provider's quota. Set it below whatever the project currently documents, and re-read that page before each planning cycle. Free quotas move.

Artifact two: the handoff card

Paste this block into the team wiki and copy it per run. The card is the artifact a reviewer reads.

### Run card r-2026-10-18-a
- Owner: morgan
- Slot: 09:00-11:00 UTC
- Goal (one sentence): add retry jitter to the fetch wrapper
- Model: as listed by the project today
- Files touched: src/fetch.ts, test/fetch.test.ts
- Test command: npm test -- fetch
- Ledger line: recorded, 8,412 tokens
- Open risk: jitter default may break timing test
- Reviewer verdict: pending

A card with a missing test command is a block. A card with no ledger line is a block. A card with two goals is split before review, not after.

Artifact three: a bench smoke test

The free server option deserves the same suspicion as any demo host. Four cheap checks answer the only question that matters: can the team rely on it for the next two hours?

#!/usr/bin/env bash
# bench/smoke.sh — cheap checks before a shift on the free server
set -euo pipefail
BASE="${BENCH_BASE_URL:?set BENCH_BASE_URL}"

probe() {
local label="$1" want="$2" path="$3" ttl="$4" code
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time "$ttl" "$BASE$path" || true)
if [[ "$code" == "$want" ]]; then
printf '%-22s ok (%s)\n' "$label" "$code"
else
printf '%-22s FAIL (want %s, got %s)\n' "$label" "$want" "$code"
fi
}

probe reachable 200 /healthz 5
probe unauth-reject 401 /v1/models 5
probe cold-start 200 /v1/models 30
probe metrics 200 /metrics 5

The paths above are placeholders. Swap in the endpoints the project documents, keep the timings, and log the output next to the card. A cold start above thirty seconds is a scheduling fact, not a bug report.

When to use the bench, and when not to

Limits worth stating plainly

The free allowance and the free server are operator statements, not service guarantees. Quotas, model lists, and uptime can change without notice, so re-read the project page before you plan a sprint. The ledger only counts what you save to disk; a run that skips the record step is invisible to the fence. The smoke test proves reachability, not output quality.

Skip this playbook if you are the only person using the bench. A solo user needs a budget note, not three roles. Skip it too if the work touches regulated data, because a shared free host is the wrong place for that conversation.

Start with the ledger. Run it for one week before you invite a second person onto the bench.#ai #opensource #devops #tutorial #software #coding #development #engineering #inclusive #community
A One-Page SOP for a Shared Free AI Bench

I opened two terminals at 08:20 and closed them at 19:40.
One terminal held a free server box. The other held an agent writing patches.
By lunch I had three diffs and no idea which run produced which.

This is a letter to that version of me. It is also a workflow you can copy.
It uses free model access as a second reader, and a free server as the test runner.
Both are operator-supplied availability claims, not performance promises.

What the day actually cost

I lost roughly three hours re-running the loop. I could not match diffs to runs.
I lost another two hours debugging a patch a reviewer model had blessed.
The rest went to a fixture directory that changed hash between runs.

None of those failures were the model's fault. All three were missing counters.

Mistake 1: I let the harness count attempts, not me

The agent retried. The wrapper retried. The shell loop retried again.
Three layers of retry, one log file, zero timestamps. That is not observability.

Fix: write a ledger row before the first model call, not after the last.
The ledger is a TSV file. It is boring and it saved the next day.

#!/usr/bin/env bash
# Adapted example. Not run as-is; substitute your own paths and commands.
set -euo pipefail

RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
ART="./runs/$RUN_ID"
mkdir -p "$ART"
LEDGER="./ledger.tsv"

log() { printf '%s\t%s\t%s\t%s\n' "$RUN_ID" "$1" "$2" "$3" >> "$LEDGER"; }

FIXTURE_HASH="$(find fixtures -type f -print0 | sort -z \
| xargs -0 sha256sum | sha256sum | cut -d' ' -f1)"
log fixture "$FIXTURE_HASH" "frozen-input"
log attempt "1" "start"

The rule I now follow: no ledger row, no model call.
A run without a row is a rumour, even if the diff looks correct.

Mistake 2: I treated reviewer agreement as a test

I asked a model to check a patch. It said the patch looked correct.
I merged it. A unit test then failed on an off-by-one in the range check.

A reviewer model produces an opinion. Your test suite produces an exit code.
Only one of those two things can block a merge in your CI.

Fix: the reviewer sees the diff and the failing output. Nothing else.
Feed it the smallest context that still explains the defect.

# Adapted example: reviewer input is diff + failing test output only.
git diff --no-color > "$ART/patch.diff"
pytest -q 2>&1 | tail -n 40 > "$ART/test_tail.txt" || true

# Build the reviewer payload from those two files, not from the repo.
printf '%s\n' 'Review this diff against this test failure.' \
'Return JSON: {"verdict":"agree|disagree","reason":"..."}' \
'--- DIFF ---' > "$ART/review_prompt.txt"
cat "$ART/patch.diff" >> "$ART/review_prompt.txt"
printf '%s\n' '--- FAILURE ---' >> "$ART/review_prompt.txt"
cat "$ART/test_tail.txt" >> "$ART/review_prompt.txt"

Then compare two things: the reviewer verdict and the test exit code.
If they disagree, stop. A human reads the run before anything else happens.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Mistake 3: I ran the writer and the reviewer in one workspace

The writer agent wrote scratch files. The reviewer read that polluted directory.
The next run's fixture hash no longer matched the ledger, so the diff was unprovable.

Fix: separate directories, one hash gate, and a write-path allowlist.
Run the gate before any shell command executes. Not after.

# 1) Plan must be small, typed, and declare its write paths.
jq -e '
(.steps | length) > 0 and
(.steps | length) <= 5 and
has("write_paths") and
all(.steps[]; (.cmd | type) == "string")
' plan.json > /dev/null

# 2) Write paths must stay inside the allowed trees.
jq -r '.write_paths[]' plan.json | while read -r p; do
case "$p" in
./src/*|./tests/*) ;;
*) echo "refusing path: $p" >&2; exit 3 ;;
esac
done

# 3) Fixture pack must be unchanged since the ledger row was written.
NOW="$(find fixtures -type f -print0 | sort -z \
| xargs -0 sha256sum | sha256sum | cut -d' ' -f1)"
[ "$NOW" = "$FIXTURE_HASH" ] || { echo "fixture drift" >&2; exit 4; }

The loop that survived the week

1. Freeze the fixture pack and log its hash before anything else.
2. Write the plan as JSON. Validate it with jq and the allowlist gate.
3. Run the test suite on the free server option, capture exit code and tail.
4. Send only the diff and the failure tail to the free model access pass.
5. Compare verdict with exit code. Disagreement halts the run.
6. Append one ledger row per step, including wall-clock seconds.

Six steps produce a receipt you can replay on Friday. That is the whole point.

When free capacity is the wrong tool

Free access is availability, not a throughput guarantee. I plan short bursts, not deadlines.
I also assume nothing about persistence. Artifacts come back to my machine immediately.

Three rows are judgement calls. Two rows are hard limits, not preferences.

Limitations and who should skip this

Free model access and the free server option are operator-supplied claims. Verify terms yourself.
I did not measure throughput, and I would not schedule a release on it.

Skip this workflow if your tests need production credentials or regulated data.
Skip it if you need contractual uptime or data residency guarantees.
Skip it if a human reviewer cannot read the ledger before merge.

What I would tell 08:20-me

Write the counter before you write the prompt. Write the hash before the diff.
When the reviewer and the test disagree, believe the test and call a human.

If you want to try this, run the six-step loop once against free model access and the free server option. Keep the ledger even if you stop using both tomorrow.#ai #testing #devops #productivity #software #coding #development #engineering #inclusive #community
Letter to 08:20-Me: Stop Auditing the Agent, Audit the Counters

Imagine the same prompt, the same repository, and the same command, run on two different afternoons. On the first afternoon the agent finishes quickly and the test suite is green; on the second it edits four extra files and leaves two tests red. Nothing in the repository changed between those runs, and that is precisely why the second outcome is difficult to defend in a code review.

Most comparisons of free AI coding tiers begin and end with price. The question that actually changes your workflow is whether a tier behaves consistently enough that you can build a habit around it, because a cheap tool with a wide outcome spread quietly spends your review time instead. This article converts that spread into a small ledger you can run in one sitting, and it retires four claims that developers repeat whenever tiers are compared.

The format is a myth-busting FAQ, because myths are what get repeated in chat threads while the measurements stay private. Each claim below comes with a cheap check and a corrected mental model, and none of the checks requires a paid plan to be interesting.

Myth 1: The same prompt produces the same run

Repeat a task eight times against the same commit and you will rarely get eight identical diffs. Sampling settings, tool-call ordering, network retries, and the order in which files are read all push the run in slightly different directions. A single green run is therefore an anecdote about one sample, not a property of the tier.

The corrected mental model is that you are not buying a function, you are sampling a distribution. That sounds abstract until you write the distribution down, and writing it down takes less effort than arguing about it in a pull request. Once you have rows, the conversation moves from "it worked for me" to "median acceptable, tail unacceptable".

Myth 2: A green run verifies the workflow

A passing test suite proves that one sample passed, and it says nothing about how often the next sample will pass. The failure mode is subtle: the agent's diff and the test outcome are recorded together in your head, but only the happy version survives into the team's memory. Recording both columns for every run is the entire fix.

The corrected model separates process from task. Exit code zero tells you the command finished; a test result tells you the change was acceptable; a diff count tells you how much review surface the change created. Those three signals disagree often enough that collapsing them into one boolean will mislead you.

Myth 3: The median is the number that matters

Medians are comfortable and tails are expensive, and free tiers tend to produce tails. If seven runs finish in a few minutes and the eighth takes an hour, your median looks healthy while your afternoon is gone. Anyone who has waited on a stuck job knows the average is not the number that ruins the day.

The corrected model is a tail ratio: worst run divided by median run, computed from the same ledger rows. A tier with a stable median and a large tail ratio is fine for exploration and hostile for anything with a deadline. Compute both numbers before you decide which kind of tool you are holding.

The ledger: eight runs, four columns

The harness below is deliberately boring, and you should adapt the one line that invokes your agent. It records the commit you started from, resets the worktree between runs, and appends one CSV row per attempt. Treat it as a proposal to run on your own task rather than as a benchmark result, since the numbers you get will depend on your repository.

#!/usr/bin/env bash
# variance_ledger.sh — repeat one task N times, record one row per run
set -uo pipefail

N="${N:-8}"
TASK="${TASK:-tasks/fix_import_order.md}"
OUT="ledger/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"

git rev-parse HEAD > "$OUT/base_commit.txt"
git status --porcelain > "$OUT/base_dirty.txt" # should be empty

for i in $(seq 1 "$N"); do
start=$(date +%s)
# Replace this line with your agent's non-interactive entry point.
agent run --task "$TASK" --no-interactive > "$OUT/run_$i.log" 2>&1
code=$?
end=$(date +%s)
added_deleted=$(git diff --numstat | awk '{a+=$1; d+=$2} END {print (a+0) " " (d+0)}')
printf '%s,%s,%s,%s\n' "$i" "$((end-start))" "$code" "$added_deleted" >> "$OUT/ledger.csv"
git checkout -- . >/dev/null 2>&1
git clean -fdq
done

The reset step is what makes the rows comparable, and skipping it is the most common way to poison a ledger. Running without a clean worktree means run two starts from run one's leftovers, and the whole table then measures compounding edits instead of tier behaviour. Check base_dirty.txt before you trust anything downstream.

Summarising takes a short awk script, because pulling in a dataframe library for eight rows is unnecessary ceremony.

awk -F, '{wall[NR]=$2; n=NR}
END{
for(i=1;i<=n;i++) for(j=i+1;j<=n;j++) if(wall[j]<wall[i]){t=wall[i];wall[i]=wall[j];wall[j]=t}
med=(n%2)?wall[(n+1)/2]:(wall[n/2]+wall[n/2+1])/2
printf "runs=%d min=%ss median=%ss max=%ss tail_ratio=%.2f\n", n, wall[1], med, wall[n], wall[n]/med
}' ledger/*/ledger.csv

Add the task verdict yourself, because wall time cannot tell you whether a diff was correct. A second column holding the exit status of your test command is usually enough, and it keeps the ledger honest when a fast run produces a bad patch.

Myth 4: Variance is a property of the model

Variance usually arrives from the environment wrapped around the model. A shared shell with a warm package cache, a locale that formats numbers differently, and a proxy that silently retries a request are three common sources that have nothing to do with the weights. If you change the environment between runs, you have measured the environment.

The corrected model treats the agent as one variable among several, and it forces you to hold the others fixed or record them. A warm cache and a different locale are environment variables in the literal sense, and both explain why two teams can disagree about the same tier while both are reporting honestly.

Where free access and a free server actually help

Repetition is the step everyone skips, because repetition is what costs money and occupies a machine. MonkeyCode's free model access and free server option change the arithmetic of that step, since an eight-run ledger stops being an expense and becomes an ordinary afternoon task. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The honest version of that claim is narrower than a slogan. Free access makes sampling practical at small n, and a free server keeps the loop off your laptop while you review the previous run; neither makes a tier deterministic. Check the project's own documentation for current allowances and machine characteristics, because quotas and hardware are exactly the details that go stale fastest.

Limitations worth stating plainly: eight runs is a small sample, diff size is not correctness, wall time includes network noise you cannot control, and every repetition consumes allowance you may want later. The ledger also assumes a non-interactive entry point, which not every agent exposes cleanly. Treat the output as a screen for whether a tier deserves careful measurement, not as a benchmark.

Who should not use this approach

Teams that need deterministic output, contractual latency, or regulated audit trails will not be served by sampling a free tier at all. Anyone whose task takes an hour per run should shrink the task before running any ledger. And if review capacity is the real bottleneck, variance reduction matters more than cost reduction, so pay for stability rather than collecting cheap rows.

The myth-busting habit generalises beyond any single product: write the claim down, find the cheapest observation that could refute it, and keep the rows. Eight runs and four columns will not settle a platform debate, but they will stop you from repeating a one-sample story as if it were a property. If you want to try the ledger, the free tier is enough to learn whether your own task is stable before you commit real budget to it.#ai #testing #devtools #productivity #software #coding #development #engineering #inclusive #community
Zero Cost Is Not Zero Variance: Four Myths About Repeating an AI Coding Run

Suppose an agent opens a pull request that adds a summarization endpoint. CI is green. The diff also edits the shared config loader and one auth middleware file.

That is the normal case, not the dramatic one. Agent-written PRs rarely fail review on syntax. They fail on scope.

This article gives a three-bucket triage and a reproducible audit script. The goal is a review verdict a second engineer can re-run.

Green CI proves compilation, not intent

A passing pipeline answers one question. Does the code build and satisfy existing tests?

It does not answer whether the change belongs in this PR. It does not answer whether a shared default drifted. Treat the badge as a floor, never as evidence.

An agent optimizes for the stated task. Anything adjacent to the task looks like fair game to it. Config loaders, CI workflows, and middleware sit adjacent to almost every feature.

Step 1 — Sort the diff into three buckets

Before reading logic, classify every hunk. The bucket decides how much attention the hunk earns.

The asymmetry matters. A wrong trust hunk costs a revert. A wrong revert hunk costs an incident.

Step 2 — Measure blast radius with a script

The first artifact is a scope audit. It reads the diff, not the description.

#!/usr/bin/env python3
"""diff_scope.py - flag PR changes that fall outside the stated scope."""
import fnmatch
import subprocess
import sys

SENSITIVE = [
"config/*", "**/settings.py", "**/defaults.py",
"**/auth/*", "**/middleware/*",
".github/workflows/*",
"migrations/*", "**/migrations/*",
"requirements*.txt", "poetry.lock", "package-lock.json",
]

def changed_files(base):
out = subprocess.run(
["git", "diff", "--name-only", f"{base}...HEAD"],
capture_output=True, text=True, check=True,
).stdout
return [line for line in out.splitlines() if line.strip()]

def classify(files):
flagged =
[] for path in files:
for pattern in SENSITIVE:
if fnmatch.fnmatch(path, pattern):
flagged.append((path, pattern))
break
return flagged

if __name__ == "__main__":
base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
files = changed_files(base)
print(f"files changed: {len(files)}")
for path, pattern in classify(files):
print(f"SENSITIVE {path} (matched {pattern})")

Run it against the PR branch:

python diff_scope.py origin/main

One honest caveat: fnmatch treats * as greedy across slashes. The list over-matches on purpose. A review aid should produce false positives, not silent misses.

Step 3 — Grep for the three hunk shapes that hide bugs

Path filters miss logic edits inside in-scope files. Three shapes deserve a targeted grep.

git diff -U0 origin/main...HEAD -- '*.py' \
| grep -nE '^\+.*(except Exception|except:|= None|os\.environ|default=)'

Read every hit. Swallowed exceptions turn a 500 into a 200 with an empty body. Changed defaults move behavior for every caller, not just the new one. A new os.environ read adds a deployment requirement the PR description probably omits.

Step 4 — Revert scope creep before reading logic

Revert first, review second. Reading logic inside a contaminated diff wastes attention.

Use file-level restore for files the PR never should have touched:

git checkout origin/main -- app/config/defaults.py .github/workflows/ci.yml
git commit -m "revert: out-of-scope changes, split into follow-up PRs"

Then re-run CI on the reduced diff. If the feature still passes, the reverted files were genuinely unrelated. If it fails, the agent had a hidden dependency you now know about.

That failure is the useful outcome. It converts an invisible coupling into a reviewable decision.

Step 5 — Test the contract, then the failure path

The test bucket needs two tests minimum. One for the happy contract, one for the failure the agent usually forgets.

# tests/test_summarize_contract.py - sketch; adapt imports to your app.
from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)

def test_empty_text_is_rejected():
r = client.post("/summarize", json={"text": ""})
assert r.status_code == 422

def test_upstream_timeout_is_not_a_200(monkeypatch):
def boom(_text):
raise TimeoutError("model timeout")
monkeypatch.setattr("app.summarize.call_model", boom)
r = client.post("/summarize", json={"text": "hello"})
assert r.status_code >= 500
assert "timeout" not in r.text.lower() # no internal detail leaked

The second assertion catches a specific failure. An agent that adds a global error handler often leaks exception text to the client. That is a quiet information disclosure, and it ships green.

Run only the new tests first:

pytest tests/test_summarize_contract.py -q

If they pass against the unreverted diff, re-run after the revert. Order matters.

Step 6 — Record the verdict in the PR

A verdict that lives in a chat thread is not reviewable. Write it where the next reader looks.

## Review verdict
- Files outside stated scope: app/config/defaults.py, .github/workflows/ci.yml
- Reverted: both, moving to follow-up PRs
- Kept: app/summarize.py, tests/test_summarize_contract.py
- Tests added: empty-input 422, upstream-timeout 5xx, no detail leak
- Not verified: live model latency, per-request cost, load behavior

The final line is the most important. Review states what was not checked. Silence reads as coverage.

Where free model access fits

A first pass over a large diff is mechanical work. Ask a model to list suspicious hunks, then verify each one by hand.

MonkeyCode offers free model access and a free server option, per the operator. That combination suits this narrow job: a stateless prompt that returns a candidate list. The reviewer keeps every decision, and the script remains the source of truth.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the prompt small. Paste the diff, the stated task, and the sensitive-path list. Ask for hunks that match the list, with file and line. Ignore any answer you cannot confirm in git diff.

Limitations and who should skip this

This workflow is file-path driven. It cannot see a subtle logic change inside an in-scope file, and it never replaces running the suite. A single-file agent PR will pass the scope audit and can still be wrong.

The audit also assumes a stable origin/main. Rebase-heavy branches make ...HEAD comparisons noisy.

Skip this approach in three cases. Teams that require pinned, audited model versions should not paste diffs into any hosted endpoint. Teams with strict data-residency rules should keep the diff local. Teams under a hard deadline should revert aggressively rather than attempt a full model-assisted pass.

Free-tier availability and quotas can change without notice. Confirm current limits before depending on them in a pipeline.

The takeaway

Trust the mechanical, revert the adjacent, test the failure. That ordering survives model upgrades and tool churn.

The script is thirty lines. The verdict template is five. Both are cheaper than one incident caused by a config default nobody read.

Readers who want to try the model-assisted first pass can start with the free model access and free server option, then verify every flag against the raw diff.#ai #codequality #testing #devtools #software #coding #development #engineering #inclusive #community
What to Trust, Revert, and Test in an Agent-Generated Pull Request

Automating SSL Renewals in GitHub Actions with acme.sh

When our infrastructure team inherited a legacy deployment pipeline that required manual SSL certificate renewals every 90 days, the operational debt became unsustainable. After evaluating several ACME clients, we chose acme.sh for its pure shell implementation and minimal dependencies—critical for containerized CI environments.

The Problem: Certificate Expiry as a Production Incident

Our initial approach used cron jobs on individual VMs, but this created blind spots:

• No centralized audit trail of renewal attempts
• Failed renewals went unnoticed until expiry warnings
• Inconsistent acme.sh versions across environments
• Manual intervention required during DNS validation

We needed certificate lifecycle management integrated directly into our GitHub Actions deployment pipeline.

Deliverable 1: Hardened GitHub Actions Workflow

This workflow runs on a schedule and can be triggered manually. Key hardening techniques:

name: SSL Certificate Renewal

on:
schedule:
- cron: '0 2 * * 0' # Weekly Sunday 2 AM UTC
workflow_dispatch:

concurrency:
group: ssl-renewal
cancel-in-progress: false # Never cancel mid-renewal

jobs:
renew:
runs-on: ubuntu-22.04
timeout-minutes: 15

strategy:
matrix:
domain:
- api.example.com
- app.example.com
fail-fast: false # Continue other domains on failure

steps:
- name: Checkout acme.sh pinned version
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
repository: acmesh-official/acme.sh
ref: '3.0.7' # Pin exact release
path: acme.sh

- name: Cache acme.sh account
uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2 # v4.0.0
with:
path: ~/.acme.sh
key: acme-account-${{ runner.os }}-v2

- name: Install acme.sh
run: |
cd acme.sh
./acme.sh --install --nocron --home ~/.acme.sh \
--accountemail ${{ secrets.ACME_EMAIL }}

- name: Issue certificate with DNS validation
env:
CF_Token: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CF_Zone_ID: ${{ secrets.CLOUDFLARE_ZONE_ID }}
run: |
~/.acme.sh/acme.sh --issue \
--dns dns_cf \
--domain ${{ matrix.domain }} \
--keylength ec-256 \
--server letsencrypt \
--log ~/.acme.sh/acme.log

- name: Upload certificate to secrets manager
if: success()
run: |
# Example: AWS Secrets Manager
CERT=$(cat ~/.acme.sh/${{ matrix.domain }}_ecc/fullchain.cer | base64 -w0)
KEY=$(cat ~/.acme.sh/${{ matrix.domain }}_ecc/${{ matrix.domain }}.key | base64 -w0)

aws secretsmanager update-secret \
--secret-id "ssl/${{ matrix.domain }}/cert" \
--secret-string "$CERT"

aws secretsmanager update-secret \
--secret-id "ssl/${{ matrix.domain }}/key" \
--secret-string "$KEY"

- name: Upload logs on failure
if: failure()
uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 # v4.3.0
with:
name: acme-logs-${{ matrix.domain }}
path: ~/.acme.sh/acme.log
retention-days: 30

Deliverable 2: Pre-Deployment Validation Script

This bash script runs in CI before deploying certificate updates to production:

#!/usr/bin/env bash
set -euo pipefail

# validate-cert.sh - Run in CI to enforce certificate quality gates

DOMAIN="$1"
CERT_FILE="$2"

# Gate 1: Certificate expiry must be > 30 days
EXPIRY=$(openssl x509 -enddate -noout -in "$CERT_FILE" | cut -d= -f2)
EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
NOW_EPOCH=$(date +%s)
DAYS_REMAINING=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

if [[ $DAYS_REMAINING -lt 30 ]]; then
echo "FAIL: Certificate expires in $DAYS_REMAINING days (minimum 30)"
exit 1
fi

# Gate 2: Subject Alternative Name must match domain
if ! openssl x509 -noout -text -in "$CERT_FILE" | grep -q "DNS:$DOMAIN"; then
echo "FAIL: Certificate SAN does not include $DOMAIN"
exit 1
fi

# Gate 3: Certificate chain must be complete
CHAIN_DEPTH=$(openssl crl2pkcs7 -nocrl -certfile "$CERT_FILE" | \
openssl pkcs7 -print_certs -noout | grep -c "subject=")

if [[ $CHAIN_DEPTH -lt 2 ]]; then
echo "FAIL: Certificate chain incomplete (depth: $CHAIN_DEPTH)"
exit 1
fi

echo "PASS: Certificate valid for $DAYS_REMAINING days with complete chain"

War Story: The Unpinned Action Incident

In March 2025, a supply chain compromise in a popular GitHub Action inserted credential exfiltration code. We discovered our runner logs were being uploaded to an external S3 bucket—logs that contained temporary AWS credentials.

The root cause: unpinned action versions using @main tags. After incident response, we implemented two policies:

1. Pin all actions to full commit SHAs (not tags, which are mutable)
2. Weekly Dependabot PRs to update pinned SHAs with security review

Lessons from Production

Concurrency control matters. Early versions allowed parallel renewal jobs, which triggered Let's Encrypt rate limits. The cancel-in-progress: false directive prevents workflow cancellations mid-validation.

Cache keys must version-bump on breaking changes. When we migrated from RSA to ECDSA certificates, stale cached account data caused validation failures. Explicit cache versioning (-v2 suffix) gives us a manual invalidation lever.

Matrix builds isolate blast radius. With fail-fast: false, a failure renewing one domain doesn't block others. The tradeoff: longer wall-clock time, but acceptable for weekly runs.

Conclusion

Integrating acme.sh into GitHub Actions eliminated an entire class of operational incidents. The workflow runs unattended, with Slack notifications on failure and certificate metrics exported to our observability stack.

The full validation script and workflow are production-tested across 40+ domains handling 2M+ requests daily. Certificate expiry hasn't triggered a page in 18 months.#github #devops #ci #automation #software #coding #development #engineering #inclusive #community
Automating SSL Renewals in GitHub Actions with acme.sh

My demo broke at 9:58 PM. The same URL answered at 9:40. My code did not change in between.

That gap is the whole lesson. A free server is a shared, sleeping, replaceable machine. It is not a staging environment.

Below are five myths I kept repeating out loud. Then comes a 90-second probe that turns each one into a measurement.

Myth 1: "It returned 200, so it's up"

A 200 tells you one request finished. It says nothing about the next request.

Cold containers queue, boot, then answer. The status code hides all of that work.

So what did you actually measure? The status, or the latency budget?

Correct mental model: split the question into three.

1. Liveness — the process exists.
2. Readiness — it can serve a real request.
3. Warmth — it can do that without a boot penalty.

Record connect time, time to first byte, and full body time separately. One number is not a diagnosis.

Myth 2: "My files will be there tomorrow"

Free tiers often rebuild the container between requests. Sometimes hourly. Sometimes weekly. You rarely get told which.

So don't assume persistence. Test it.

A canary is the cheapest test you have. Write a value, wait, read it back.

stamp=$(date +%s)
curl -s -X POST "$BASE/canary" -H 'Content-Type: application/json' -d "{\"v\":\"$stamp\"}"
sleep 120
curl -s "$BASE/canary?v=$stamp"

If the read-back is empty, your "temporary" upload directory is a story you tell your demo.

Correction: treat the filesystem as a cache. Keep state in an external store, or ship a seed script that rebuilds it on every boot.

Myth 3: "Free means I can retry CI forever"

Retries are cheap for you. They are not cheap for the host.

Many free tiers count requests, not successes. Worse, retry storms hit a cold service and trigger more boots.

Your pipeline gets slower. Your logs get noisier. Nothing gets fixed.

Correction: exponential backoff with jitter, a hard attempt cap, and one pre-warm request before the real one.

for i in 1 2 3; do
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 "$BASE/healthz")
[ "$code" = "200" ] && break
sleep $(( (2 ** i) + RANDOM % 3 ))
done

Myth 4: "Green locally means green remotely"

At least six things differ between your laptop and a free container:

• Timezone — logs land on the wrong day.
• Clock — token expiry and signed URLs drift.
• Env vars — one missing key kills a code path you never exercised.
• Filesystem — sometimes read-only, sometimes wiped.
• Memory ceiling — your build is fine until it isn't.
• Outbound network — allowlists and DNS behave differently.

None of these appear in a unit test. All of them appear in a demo.

Myth 5: "The health check proves demo readiness"

Most health endpoints are shallow. They return 200 while the connection pool is still warming.

A green dot means the route exists. It does not mean the first real user succeeds.

Correction: add one endpoint that touches a real dependency. Hit it twice before you present.

The probe: one script, one receipt

Here is the template I use now. It needs only bash, curl, and the Python standard library.

You supply two routes: /healthz and /canary. The second one stores and returns a small value.

#!/usr/bin/env python3
# probe.py <base_url> [sleep_seconds] — stdlib only, writes receipt.json
import datetime, json, sys, time, urllib.request

BASE = sys.argv[1].rstrip("/")
WAIT = float(sys.argv[2]) if len(sys.argv) > 2 else 0.0

def timed(path, payload=None, timeout=25):
req = urllib.request.Request(BASE + path, method="POST" if payload else "GET")
if payload:
req.add_header("Content-Type", "application/json")
t0 = time.perf_counter()
try:
with urllib.request.urlopen(req, data=payload, timeout=timeout) as r:
body = r.read().decode("utf-8", "replace")
return {"path": path, "status": r.status,
"ms": round((time.perf_counter() - t0) * 1000, 1),
"body": body[:200]}
except Exception as e:
return {"path": path, "status": None,
"ms": round((time.perf_counter() - t0) * 1000, 1),
"error": str(e)[:200]}

stamp = str(int(time.time()))
checks = [
timed("/healthz"), # possibly cold
timed("/healthz"), # hopefully warm
timed("/canary", payload=json.dumps({"v": stamp}).encode()),
]
time.sleep(WAIT)
checks.append(timed("/canary?v=" + stamp)) # canary read-back

receipt = {
"base": BASE,
"at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"wait_seconds": WAIT,
"checks": checks,
}
json.dump(receipt, open("receipt.json", "w"), indent=2)
print(json.dumps(receipt, indent=2))

Run it twice. Once cold, once after a short pause.

python3 probe.py your-app.example 0 > cold.json
python3 probe.py your-app.example 120 > warm.json

Reading the receipt

Do not eyeball the whole file. Check four fields.

Two runs prove very little about a month. Two runs prove a lot about tonight's demo.

Where a free model fits (and where it does not)

I use a model to summarize the receipt into three sentences for a shared channel. That is it.

A summary is not a gate. Never let a generated paragraph decide whether you ship.

For that write-up step I have used MonkeyCode's free model access and its free server option, so the probe and the summary live in one place. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Run the probe yourself regardless of host. The numbers are yours, not mine.

Decision table: is a free server enough?

Limitations and who should not use this

The probe measures only what you instrument. It cannot see the host's scheduler, quotas, or eviction policy.

It also cannot promise anything about next month. Free tiers change shape without notice.

Skip this approach entirely if you need an SLA, handle regulated data, or run stateful services. Skip it if a mid-demo redeploy would cost you the deal.

Also skip it if you cannot write the canary endpoint. Without it, you are guessing.

What I changed

I stopped calling the free box "staging". I call it a demo target with a timer.

The receipt file now lives in the repo. Every demo starts with two runs and one glance at the table.

What would your 9:58 PM failure look like, if you measured it at 9:40?#devops #testing #webdev #ai #software #coding #development #engineering #inclusive #community
Cold Starts, Lost Files, and Green Dots: Five Myths About Using a Free Server as Staging

The green run that almost fooled me

On Wednesday I moved a flaky test off my laptop and onto a remote box, and it passed 200 times in a row. For about an hour I believed I had accidentally fixed something, which is a dangerous thing to believe at 2 a.m. The test was a concurrent append to a small ledger file, and on my laptop it failed roughly one run in fifteen. So what did the move actually change — the code, the data, or just the shape of the machine underneath it?

Nothing in the code changed. That is the whole point of this post: the failure rate of a timing-sensitive bug is a property of the environment, not of your test name. If you only ever run the repro on one machine, you are measuring that machine's scheduler, not your program's correctness.

What I actually did in the first twelve hours

I stopped trying to "reproduce the bug faster" and started trying to reproduce the sensitivity instead. Three things happened, in this order:

1. I wrote down the exact conditions of the original failure (laptop, on battery, browser open, test file unmodified for six days).
2. I ran the same test 200 times on the remote box and got zero failures, which I initially misread as "not reproducible."
3. I built a sweep that runs the test under deliberately worse scheduling conditions, because the remote box was simply too forgiving.

That third step is the one I would repeat. The rest was mostly me arguing with my own assumptions.

The artifact: a scheduler-sensitivity sweep

Here is the harness, stripped down. It does not try to find the bug; it tries to find the profile under which the bug is willing to appear.

#!/usr/bin/env bash
# sweep.sh — run one test under several scheduling profiles
set -u
RUNS="${RUNS:-200}"
TEST_CMD="${TEST_CMD:-python -m pytest tests/test_ledger.py::test_concurrent_append -q}"

run_profile () {
local name="$1"; shift
local fails=0
for _ in $(seq 1 "$RUNS"); do
if ! "$@" >/dev/null 2>&1; then fails=$((fails + 1)); fi
done
printf '%-14s %s/%s failed\n' "$name" "$fails" "$RUNS"
}

run_profile "default" bash -c "$TEST_CMD"
run_profile "one-core" taskset -c 0 bash -c "$TEST_CMD"
run_profile "tight-gil" env PY_SWITCH_INTERVAL=0.000001 bash -c "$TEST_CMD"
run_profile "cpu-starved" cpulimit -l 15 -- bash -c "$TEST_CMD"

The tight-gil profile needs one line of glue in conftest.py, because there is no environment variable for sys.setswitchinterval out of the box:

# conftest.py
import os
import sys

if (raw := os.getenv("PY_SWITCH_INTERVAL")):
sys.setswitchinterval(float(raw))

Then, before trusting any of it, I check that the profile actually took effect on this machine. A profile that silently does nothing is worse than no profile at all.

# probe.py — is this box coarse or fine grained right now?
import statistics
import sys
import threading
import time

def median_gap_ms(n: int = 2000) -> float:
hits: list[int] =

[] def worker() -> None:
for _ in range(n):
hits.append(time.perf_counter_ns())

t = threading.Thread(target=worker)
t.start()
worker()
t.join()
gaps = [b - a for a, b in zip(hits, hits[1:])]
return statistics.median(gaps) / 1e6

print(f"switchinterval={sys.getswitchinterval() * 1e3:.3f}ms "
f"median_gap={median_gap_ms():.3f}ms")

My result table looks like this shape, and yours will have different numbers in it:

profile failures
-----------------------
default <n1>/200
one-core <n2>/200
tight-gil <n3>/200
cpu-starved <n4>/200

The shape is the finding, not the digits. If default is clean and one constrained profile is not, your bug is scheduling-sensitive and your fast machine has been lying to you by omission.

Where free model access and a free server actually helped

The remote box here was MonkeyCode's free server option, and the model I used for a second opinion was their free model access; both are operator-supplied availability claims, and I am describing my own use rather than a spec. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two uses were genuinely useful, and a third was a trap:

• Useful: drafting additional budget profiles for the sweep, which I then reviewed line by line before running anything.
• Useful: running the 800-run sweep off my laptop, so a sleeping lid or a browser tab could not contaminate the timing I was trying to measure.
• A trap: asking a model to "fix" the race directly. It produced a plausible lock that serialized the whole ledger and made the test pass by deleting the concurrency it was supposed to verify.

That last point matters more than the first two. Free access to a model is cheap; free access to a critic that does not get to commit is the actual workflow.

What broke, and what I would not repeat

• Wall-clock timestamps in the test assertions. They passed on the slow box and failed on the fast one, for reasons unrelated to the bug.
• Treating a clean 200-run sweep as proof. Zero failures under one profile is one data point about one scheduler.
• Running the sweep in CI first. CI is shared and noisy, so I could not tell whether a failure came from the bug or a neighbor.

I also had to stop myself from quoting the sweep's timings as performance data. The free server may be shared, so latency numbers from it describe contention on that host and nothing about your code — a lesson I apparently need to relearn periodically.

A short decision table

Who should not do this

Do not use this sweep if your defect is deterministic, because you will burn a day measuring a scheduler that was never involved. Do not use it if you need real throughput numbers, since a shared free host cannot give them to you. And do not run it against anything holding production credentials, because a remote box you did not provision is the wrong place for secrets.

What I would repeat on Monday

Build the sensitivity sweep before the fix, run it somewhere your laptop cannot interfere, and keep the model in the reviewer's chair instead of the committer's chair. If a green run on a new machine ever reassures you, ask the obvious question first: did the bug go away, or did the timing? If you run the sweep yourself, I would like to know which profile finally reproduced your bug — that is the setting I want to steal.#debugging #testing #python #devops #software #coding #development #engineering #inclusive #community
48-Hour Field Notes: My Race Condition Vanished on a Free Server — It Was Just Slower

Most documentation guardrails try to police the draft, but this one decides when a section stops being draftable at all. The mechanism is a four-trigger escalation table plus a small git-trailer gate, and the payoff is that reviewers spend attention on the few sections that carry real risk. Everything below runs on repo-native inputs: headings, anchors, git grep hits, commit churn, and one hand-maintained impact list.

The scope is deliberate. This workflow covers reference and how-to pages that a model may partially regenerate, and it assumes you already have CI that runs on pull requests. It does not attempt to decide whether a sentence is true, because that question needs a source registry and a different gate entirely.

Why ownership beats per-run gating

A draft gate answers one question per generation run: is this output acceptable right now? An ownership table answers a different question that survives across runs: is this section allowed to be generated at all by anyone other than a named human? The first is a filter, and the second is a boundary, and you need both because filters are forgettable while boundaries are enforceable.

The practical failure mode is silent promotion. A model drafts an auth section in March, a human fixes two sentences in April, and by June nobody remembers that the paragraph about token expiry was machine-authored text nobody ever verified. A trailer on the commit plus a per-section owner marker makes that history greppable instead of tribal.

Step 1: Name four escalation triggers

A trigger is only useful if a script can evaluate it without asking a human for an opinion. I use four, each mapped to a command you already have installed, and each with a threshold you will need to tune to your repo size. Write the thresholds down in the script, not in a wiki page, so a threshold change shows up in code review.

1. Reach — the section anchor appears in three or more source or example files. High reach means a wrong sentence propagates into user code.
2. Churn — the section's file changed in three or more commits in the last ninety days. Churn means the underlying behavior is still moving and the text tracks it.
3. Impact — the anchor appears on a hand-maintained list of sections that were cited in a support ticket or incident retrospective. This is the only subjective input, and keeping it in a reviewed JSON file keeps it small.
4. Irreversibility — the section covers install, authentication, security, migration, or billing. A reader who follows a wrong irreversible step cannot undo it cheaply, so this trigger escalates alone.

The rule is then mechanical: irreversibility always escalates, and any two of the other three escalate. One trigger alone leaves the section model-draftable, which keeps the human-owned set small enough that it actually gets read.

Step 2: Score sections from repo data

The scorer walks the docs tree, extracts headings as anchors, and evaluates each trigger. It writes a single JSON manifest that later steps consume, so the gate never re-implements the thresholds. Treat this as a reference implementation and profile it before wiring it into a large corpus.

#!/usr/bin/env python3
"""tools/doc_owner_score.py — classify doc sections as model-draftable or human-owned."""
import json
import re
import subprocess
from pathlib import Path

ROOT = Path(__file__).resolve().parents
[1]DOCS = ROOT / "docs"
IMPACT_FILE = DOCS / "impact.json" # {"anchors": ["auth-tokens", "billing-proration"]}
IRREVERSIBLE = ("install", "auth", "security", "migration", "billing")

REACH_MIN, CHURN_MIN, CHURN_DAYS = 3, 3, 90
HEADING = re.compile(r"^(##|###)\s+(.+)$", re.M)

def git(*args: str) -> str:
return subprocess.run(
["git", *args], cwd=ROOT, capture_output=True, text=True, check=False
).stdout

def slug_of(title: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")

def code_refs(slug: str) -> int:
hits = git("grep", "-l", "-F", f"#{slug}", "--", "src", "examples")
return len([h for h in hits.splitlines() if h.strip()])

def churn(path: Path, slug: str) -> int:
"""Count recent commits whose patch mentions this anchor. Cache this in CI."""
count = 0
for sha in git("log", f"--since={CHURN_DAYS}.days", "--format=%H", "--", str(path)).split():
patch = git("show", "--unified=0", "--format=", sha, "--", str(path)).lower()
count += slug in patch
return count

def classify(slug: str, path: Path, impact: set[str]) -> dict:
flags = {
"reach": code_refs(slug) >= REACH_MIN,
"churn": churn(path, slug) >= CHURN_MIN,
"impact": slug in impact,
"irreversible": any(word in slug for word in IRREVERSIBLE),
}
secondary = sum(flags[k] for k in ("reach", "churn", "impact"))
owner = "human" if flags["irreversible"] or secondary >= 2 else "model"
return {"anchor": slug, "file": str(path.relative_to(ROOT)), "owner": owner, "flags": flags}

def main() -> None:
impact = set(json.loads(IMPACT_FILE.read_text()).get("anchors", []))
sections =
[] for path in sorted(DOCS.rglob("*.md")):
for match in HEADING.finditer(path.read_text()):
sections.append(classify(slug_of(match.group(2)), path, impact))
DOCS.joinpath("ownership.json").write_text(
json.dumps({"sections": sections}, indent=2) + "\n"
)
human = [s["anchor"] for s in sections if s["owner"] == "human"]
print(f"{len(sections)} sections, {len(human)} human-owned")
for anchor in human:
print(f" human: {anchor}")

if __name__ == "__main__":
main()

The churn function runs git show once per commit, which is acceptable for a nightly job and unacceptable for a pre-commit hook. Cache the patch text by SHA if you need it on every push, and expect the first run on a large history to take noticeably longer than later runs.

Step 3: Mark ownership inline, not out of band

The manifest tells CI what the rules produced, but a reviewer reading the Markdown should not have to open a second file. Put a single HTML comment above each section that states its owner, and let the manifest be a derived artifact that the gate cross-checks. Comments render invisibly on the published page and survive Markdown linting.

<!-- owner:human anchor:auth-tokens -->
## Rotating API tokens

<!-- owner:model anchor:list-supported-regions -->
## Supported regions

If the manifest and the inline marker disagree, the gate should fail rather than pick a winner. A disagreement means someone hand-edited the manifest or forgot to re-score after a large content change, and both are worth a human look before a model touches the page again.

Step 4: Gate model-drafted commits with a trailer

Ownership only binds if the commit that violates it cannot merge. Tag machine-assisted commits with a Drafted-By trailer, then have the gate map each changed line range in a pull request back to its nearest preceding owner marker. Git's --trailer flag handles the tagging on recent versions, and older clients can append the same two lines to the message body by hand.

git commit -m "docs: regenerate model-owned reference sections" \
--trailer "Drafted-By: model" \
--trailer "Draft-Run: 2026-10-17T09:00Z"

#!/usr/bin/env python3
"""tools/doc_gate.py — fail a model-drafted commit that edits human-owned lines."""
import re
import subprocess
import sys
from pathlib import Path

MARK = re.compile(r"<!--\s*owner:(human|model)\b")
HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@")

def drafted_by() -> str:
out = subprocess.run(
["git", "log", "-1", "--format=%(trailers:key=Drafted-By,valueonly)"],
capture_output=True, text=True, check=False,
).stdout.strip().lower()
return out

def owner_at(path: Path, line_no: int) -> str:
marks = [(path.read_text()[: m.start()].count("\n") + 1, m.group(1)) for m in MARK.finditer(path.read_text())]
current = "human" # unmarked sections default to human
for mark_line, owner in marks:
if mark_line <= line_no:
current = owner
return current

def main() -> int:
if drafted_by() != "model":
print("human-authored commit, skipping ownership gate")
return 0

diff = subprocess.run(
["git", "diff", "--unified=0", "origin/main...HEAD", "--", "docs"],
capture_output=True, text=True, check=False,
).stdout

violations, path, pending = [], None, None
for line in diff.splitlines():
if line.startswith("+++ b/"):
path = Path(line[6:])
elif (hunk := HUNK.match(line)):
pending = int(hunk.group(1))
elif pending is not None and path is not None and line[:1] in ("+", "-"):
if owner_at(path, pending) == "human":
violations.append(f"{path}:{pending}")

if violations:
print("model-drafted commit touched human-owned sections:")
for spot in sorted(set(violations)):
print(f" {spot}")
return 1
return 0

if __name__ == "__main__":
sys.exit(main())

name: doc-ownership
on:
[pull_request]jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- name: Ownership gate
run: python3 tools/doc_gate.py

The gate defaults unmarked sections to human, which means adding a new page requires an explicit owner:model decision before a batch job may rewrite it. That default is intentional and slightly annoying, because the annoying version is the one that survives a busy week.

Step 5: Draft only the model-owned set

The nightly job reads docs/ownership.json, filters to owner == "model", and regenerates only those sections from their source inputs. I run that batch against MonkeyCode's free model access, which keeps the marginal cost of a full regeneration pass at zero for a repo of this size. The scheduler itself lives on the free server option rather than a paid runner, so a failed pass costs nothing but a retry. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Because a free tier carries no availability promise, the job is written to be idempotent and restartable: it writes to a branch, never to main, and it re-reads the manifest on every run so a section promoted by the scorer cannot be regenerated. I cannot quote quotas or guarantees here, and you should not assume any before scheduling a job that must run at a fixed hour.

python3 tools/doc_owner_score.py
git checkout -B docs/auto-refresh
python3 tools/draft_model_sections.py --manifest docs/ownership.json --out docs/
git commit -am "docs: regenerate model-owned reference sections" \
--trailer "Drafted-By: model" \
--trailer "Draft-Run: $(date -u +%Y-%m-%dT%H:%MZ)"

The escalation table at a glance

When a section flips from model to human, the flip is the deliverable. Open a small pull request that only changes the marker comment and regenerates the manifest, so the promotion gets its own reviewable moment instead of hiding inside a large regeneration diff.

Limitations and who should skip this

Four triggers are a heuristic, not a measurement of correctness, and a section can be wrong while scoring zero on every axis. The scoring script assumes anchors are stable, so renaming headings resets churn history and quietly un-escalates sections that deserve attention. Repos with fewer than roughly twenty doc pages will find the table overhead larger than the review time it saves.

Skip this if every sentence in your docs already needs a named human author for compliance reasons, because then the whole tree is human-owned and the gate degenerates into an always-fail check. Skip it too if you have no pull-request CI, since a trailer nobody enforces is just a commit message.

The piece worth adapting first is the trigger thresholds, not the code. Lower REACH_MIN to two on a small public API surface and watch how many sections flip to human before you automate anything downstream.#documentation #devops #ai #githubactions #software #coding #development #engineering #inclusive #community
When a Model-Drafted Doc Section Earns a Human Owner: An Escalation Ladder and a Trailer Gate

The swap everyone makes

You have a paid endpoint and a bill you dislike. A free endpoint appears. You change one line, and the invoice drops to zero. So why does the work feel slower two weeks later?

That is the myth I keep meeting. Free model access moves the bill. It rarely deletes it.

Free model access and a free server option are the two availability claims behind this post. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I will not quote quotas, hardware, or latency numbers I have not verified. You measure your own.

Myth 1: Free means the cost is zero

The invoice is zero. The cost moved somewhere else.

• Attempt cost. Every retry is another request.
• Wall-clock cost. A queue you wait in is still time spent.
• Human cost. Someone re-runs the job at 2am.
• Opportunity cost. You stop tuning because "it's free anyway."

Add those four lines and the endpoint stops looking free. It is cheap in dollars and expensive in attempts. Do you know your attempts per successful task today?

Myth 2: It is the same model, only cheaper

This one bites hardest. The same family name is not the same serving path.

Rate limits and timeouts can differ per deployment. So can context handling and truncation behavior. A familiar model name tells you nothing about the endpoint behind it.

Do not assume parity. Probe it. Send one fixed prompt many times. Record latency, output length, and whether the answer is usable.

If output length shrinks across a long run, that is a serving difference. It is not a bad afternoon. Corrected model: treat a free endpoint as a different deployment, not a discount on the same one.

Myth 3: It is free, so fan it out

Fan-out multiplies attempts, not just traffic. Ten parallel CI jobs become one queue.

Queues become timeouts. Timeouts become retries. Retries become duplicate work. Now the free pipeline is slower than the paid one it replaced.

My rule is blunt. Fan-out is real when attempts per success stays near one. Above that, your parallelism is fake. You are only moving load around.

Myth 4: A free server means I can skip setup scripts

No. It means setup has to be scripted, because yesterday's state may not exist.

If your runbook lives in your head, a fresh box will expose it. That is not a free-tier problem. That is a reproducibility problem you were hiding.

Corrected model: free compute rewards scripts and punishes memory.

Build a 40-line attempt ledger

Here is the artifact. It wraps any OpenAI-compatible endpoint. It logs every attempt as JSONL, so you can count real work units.

Save it as attempt-ledger.sh. Set ENDPOINT, MODEL, and TASKS before running. Three attempts per task is the retry ceiling.

#!/usr/bin/env bash
# attempt-ledger.sh - log every attempt, not just the happy path.
set -uo pipefail

ENDPOINT="${ENDPOINT:?set ENDPOINT to your base URL}"
MODEL="${MODEL:?set MODEL}"
TASKS="${TASKS:-20}"
LEDGER="${LEDGER:-attempts.jsonl}"

: > "$LEDGER"

for i in $(seq 1 "$TASKS"); do
attempt=0
ok=0
while [ "$attempt" -lt 3 ] && [ "$ok" -eq 0 ]; do
attempt=$((attempt + 1))
start=$(date +%s%3N)
code=$(curl -s -o /tmp/ledger_out.json -w '%{http_code}' \
-X POST "$ENDPOINT/chat/completions" \
-H 'Content-Type: application/json' \
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with the single word: ok\"}]}")
end=$(date +%s%3N)
[ "$code" = "200" ] && ok=1
printf '{"task":%d,"attempt":%d,"http":%s,"ms":%d,"ok":%d}\n' \
"$i" "$attempt" "$code" "$((end - start))" "$ok" >> "$LEDGER"
done
done

echo "wrote $LEDGER"

Now summarize it. This jq program prints the numbers worth arguing about.

jq -s '
def p(q): (map(.ms) | sort) as $s | $s[((($s | length) - 1) * q) | floor];
([.[] | select(.ok == 1)] | length) as $ok |
{
logical_tasks: ([.[].task] | unique | length),
total_attempts: length,
successes: $ok,
attempts_per_success: (if $ok == 0 then null else (length / $ok) end),
p50_ms: p(0.50),
p95_ms: p(0.95)
}
' attempts.jsonl

What follows is an illustrative shape only. These are not measured numbers from any product. Your run will differ.

{"logical_tasks":20,"total_attempts":27,"successes":20,"attempts_per_success":1.35,"p50_ms":1180,"p95_ms":4300}

How to read those numbers

• attempts_per_success = 1.0? The endpoint behaves like a paid one for this workload.
• 1.0 to 1.5? Acceptable for offline batch work.
• Above 1.5? You are paying in time. Fix payload size or concurrency.
• successes = 0? Stop. That is a configuration bug, not a free-tier bug.

A decision table keeps the switch honest.

What the ledger cannot see

Honesty matters more than a clean table.

• It does not measure answer quality. Read some outputs yourself.
• It does not measure your own re-run time. Add a column if you want it.
• It does not prove persistence. A server that lived yesterday may not live tomorrow.
• It does not tell you where your data goes. Ask before sending sensitive prompts.

Who should skip this approach

• Teams with data-residency constraints they cannot verify.
• Products with hard latency budgets and no fallback path.
• Jobs that keep state in memory or on local disk between runs.
• Teams that will not read a ledger file. The number is the whole point.

The corrected mental model is simple. Free model access trades dollars for attempts, queues, and attention. Free servers trade permanence for setup discipline. Neither is a discount on a system you refused to measure. If you run this ledger against MonkeyCode's free model access, tell me your attempts-per-success. That is the only comparison worth making.#ai #devops #testing #productivity #software #coding #development #engineering #inclusive #community
Myth FAQ: Free Model Access Moves the Bill, It Doesn't Delete It

Detect Silent Model Drift With a Stdlib-Only Golden Set Harness

A hosted model can change between two of your deploys, and nothing in your lockfile moves. Your unit tests still pass, your container still builds, and your users still get slightly different JSON than they got last Tuesday. The practical fix is not a bigger test suite; it is a frozen golden set, a deterministic scorer, and an append-only ledger that fails the run when the pass rate drops below a floor.

This article builds exactly that, using only the Python standard library. It runs on a free-tier host, it costs one file of input data, and it answers a single narrow question: did the contract my product depends on stop holding?

Why ordinary tests miss model drift

Application tests assert on code you control. A model response is an input from outside your repository, and it arrives as free text that happens to be parseable.

When a provider adjusts decoding defaults, swaps capacity, or routes you to a different checkpoint, the shape of the output can shift while the API status code stays 200. Nothing throws. The failure surfaces later, as a support message about a field that used to be filled in.

A golden set inverts that. Instead of asserting on the current behavior, it asserts on a small number of hand-checked examples that were correct on a specific day, and it keeps the receipts.

Step 1: Freeze a small golden set as JSONL

The input file is plain JSONL, one contract per line. Keep it small on purpose: 20 to 40 cases is enough to catch a formatting or label flip, and it fits inside a free-tier request budget.

{"id":"invoice_date","input":"Invoice paid 2026-03-04, net 30 days.","expect":{"label":"invoice","fields":{"due_date":"2026-04-03"}}}
{"id":"refund_request","input":"I want my money back for order 8812, it never arrived.","expect":{"label":"refund","fields":{"order_id":"8812"}}}
{"id":"no_action","input":"Thanks, the docs answered my question.","expect":{"label":"none","fields":{}}}

Every line has three parts: an id you will grep for in the ledger, the raw input you send, and the expect object you compare against. Do not paste generated examples into this file. If you did not read the output and agree with it, it does not belong in the baseline.

Step 2: Write a transport adapter that assumes nothing

The runner below assumes an OpenAI-style /chat/completions request body. That assumption is isolated in one function so you can replace it when your endpoint differs.

#!/usr/bin/env python3
"""golden_run.py - regression run against a chat endpoint. Stdlib only."""
import json, os, sys, time, hashlib, pathlib, urllib.request

BASE = os.environ["MODEL_BASE_URL"].rstrip("/")
KEY = os.environ["MODEL_API_KEY"]
MODEL = os.environ["MODEL_ID"]
GOLDEN = pathlib.Path(os.environ.get("GOLDEN", "golden.jsonl"))
LEDGER = pathlib.Path(os.environ.get("LEDGER", "ledger.jsonl"))

SYSTEM = (
"You are a strict extractor. Return ONLY minified JSON shaped as "
'{"label": string, "fields": object}. Omit absent fields. '
"No prose, no markdown fences."
)

def call(text, timeout=30):
body = json.dumps({
"model": MODEL,
"temperature": 0,
"messages": [{"role": "system", "content": SYSTEM},
{"role": "user", "content": text}],
}).encode()
req = urllib.request.Request(
f"{BASE}/chat/completions", data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {KEY}"},
)
with urllib.request.urlopen(req, timeout=timeout) as r:
payload = json.loads(r.read())
return payload["choices"][0]["message"]["content"]

Keep temperature at zero when the endpoint supports it. If the parameter is rejected or ignored, note that in your README rather than deleting it, because a later comparison depends on knowing what you actually sent.

Step 3: Score the contract, not the prose

Scoring is exact after light normalization: trim, lowercase, collapse whitespace. That is deliberately unforgiving, because the goal is to detect change, not to grade writing.

def norm(v):
return " ".join(v.strip().lower().split()) if isinstance(v, str) else v

def score(case, raw):
try:
got = json.loads(raw)
except json.JSONDecodeError:
return {"case": case["id"], "ok": False, "why": "invalid_json"}
want = case["expect"]
if norm(got.get("label")) != norm(want["label"]):
return {"case": case["id"], "ok": False, "why": "label",
"got": got.get("label"), "want": want["label"]}
fields, gotf = want.get("fields", {}), (got.get("fields") or {})
for k, v in fields.items():
if norm(gotf.get(k)) != norm(v):
return {"case": case["id"], "ok": False, "why": f"field:{k}",
"got": gotf.get(k), "want": v}
return {"case": case["id"], "ok": True, "why": "pass"}

Two failure reasons are distinguishable here and that distinction is the whole point: invalid_json is a format drift, label or field:* is a content drift.

Step 4: Append every run to a ledger and exit non-zero

The ledger is one JSON object per line, append-only, with a run_id derived from the system prompt, model id, and base URL. Change any of those and the id changes, which keeps old and new comparisons honest.

def run():
run_id = hashlib.sha256(
(SYSTEM + MODEL + BASE).encode()).hexdigest()[:12]
rows =
[] for line in GOLDEN.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
case = json.loads(line)
raw, err = "", None
for attempt in range(3):
try:
raw = call(case["input"])
break
except Exception as exc: # timeout, 429, DNS, 5xx
err, raw = repr(exc), ""
time.sleep(2 ** attempt)
if not raw:
rows.append({"case": case["id"], "ok": False,
"why": "transport", "err": err})
continue
r = score(case, raw)
r["raw"] = raw[:400]
rows.append(r)
total, passed = len(rows), sum(r["ok"] for r in rows)
rate = round(passed / total, 3) if total else 0.0
record = {"run_id": run_id, "ts": int(time.time()), "total": total,
"passed": passed, "rate": rate, "rows": rows}
with LEDGER.open("a") as fh:
fh.write(json.dumps(record) + "\n")
print(f"{passed}/{total} ({rate:.0%}) run_id={run_id}")
floor = float(os.environ.get("PASS_FLOOR", "0.9"))
return 0 if total and rate >= floor else 1

if __name__ == "__main__":
sys.exit(run())

Exit codes matter more than dashboards when you are alone. Non-zero from this script can page you, block a deploy, or simply stop a scheduled job from marking the day green.

Step 5: Run it on a schedule without paying for a scheduler

On a free-tier host, cron is the cheapest scheduler you have. Secrets stay in a .env file that the job sources, so the crontab itself contains no keys.

# /srv/app/run-golden.sh
set -euo pipefail
cd /srv/app
set -a; . ./.env; set +a
python3 golden_run.py >> /var/log/golden.log 2>&1

# off-peak, once a day; adjust to the hour your host is quietest
17 4 * * * /srv/app/run-golden.sh

Compare the last two runs with one command. The jq filter prints only the cases that flipped.

tail -n 2 ledger.jsonl \
| jq -c '{run_id, ts, rate, fails: [.rows[] | select(.ok == false) | .case]}'

Free-tier containers often sleep between requests. A 30-second timeout and a three-attempt backoff absorb most cold-start noise, and the transport reason keeps that noise out of your content-drift count.

Reading the ledger: a decision table

A single bad run is not a verdict. Two consecutive runs with the same failure set is the threshold worth acting on.

Limitations and who should skip this

Exact-match scoring is brittle by design and wrong for open-ended generation. Do not point this harness at a summarizer, a chat assistant, or anything whose value is in the phrasing.

A 20-to-40 case set cannot detect rare failures, and it gives no statistical confidence. Teams that need eval coverage, graders, or per-release reports should use a real evaluation platform instead.

The adapter is single-turn. Agents that call tools across multiple round trips need a different harness, or at least a recorded-transcript replay layer in front of this one.

Finally, the runtime budget is real. Free-tier quotas, sleep windows, and timeouts change; the ledger tells you when they did, but it will not make them go away.

Where MonkeyCode fits in this loop

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness is endpoint-agnostic, which is the point. In my own loop, MonkeyCode is the endpoint behind MODEL_BASE_URL and the host that runs the cron job, because its free model access and free server option keep a nightly regression run off the invoice while the project is still pre-revenue. The operator describes the free tier as including a substantial token allowance — on the order of ten million tokens — plus a free server option; verify the current terms on the project page before you size a golden set against it, since quotas and limits move faster than articles do. The project is open source, so the transport adapter is the only part you may need to touch.

If the loop is useful, the smallest honest start is one line in golden.jsonl: take a real failure your users already reported, freeze the correct output, and let the ledger watch it for you.#ai #testing #opensource #devops #software #coding #development #engineering #inclusive #community
Detect Silent Model Drift With a Stdlib-Only Golden Set Harness

A generated patch rarely fails review because of formatting, naming, or even local correctness. It fails because the reviewer cannot reconstruct the failure it was meant to fix, the files it read, and the decision it quietly made. A senior engineer paired on such a patch will often ignore the diff for the first ten minutes and walk a fixed ladder of questions instead. Once those answers are written down, the patch becomes either clearly acceptable or clearly wrong, and the debate collapses into a record.

The ladder matters more than the lint

Lint, type checks, and unit tests answer questions about the code exactly as written. They say nothing about the reasoning that produced the change, which is where generated patches go wrong most often. A patch can pass every check and still edit a module nobody asked it to touch, or hard-code an assumption that only held inside the prompt. The review therefore needs two distinct gates: a mechanical gate for the artifact, and a conversational gate for the intent behind it.

The five questions, in order

1. Reproduction first. Can the failure be reproduced from a clean checkout without the patch? Acceptable evidence is a command, an exit code, and one log line. If the honest answer is "probably", the session ends before the diff is opened.
2. Read set second. Which files, tables, or endpoints does the patch read outside the module it edits? Acceptable evidence is a grep, a trace, or an explicit import list. This catches the most common dead end: a patch that solves the stated problem by reaching into three unrelated packages.
3. Contract third. Which existing behavior does the patch promise to preserve, and where is that promise tested? Acceptable evidence is the name of an existing test that still passes. Silence here usually means the patch is a rewrite wearing a bugfix costume.
4. Rollback fourth. If the patch turns out wrong in production, what single command or flag reverses it? Acceptable evidence is the revert path itself, not a plan to write one later.
5. Owner fifth. Who answers for this decision in six months, and where is that recorded? Acceptable evidence is a name plus a file, usually the same decision record the session produces.

Questions one and two eliminate most candidates within minutes. Questions three through five decide whether the survivor is mergeable today or needs a follow-up ticket instead.

Running the session as a reproducible artifact

1. Scaffold the session directory and freeze the candidate. Copy the patch and hash it, so the reviewed bytes cannot drift during the conversation.
2. Record the base revision. Without it, no later reader can tell which tree the patch was generated against.
3. Answer the ladder in a machine-readable file. Free text in a chat thread is not a decision record; JSON or YAML is.
4. Gate on completeness before anyone runs the tests. An unanswered question should stop the pipeline, not scroll past it.

# questions.yml - the ladder as data
- id: q1_repro
text: "Can the failure be reproduced from a clean checkout without the patch?"
evidence: "command + exit code + one log line"
- id: q2_readset
text: "Which files or tables does the patch read outside the edited module?"
evidence: "grep output or trace excerpt"
- id: q3_contract
text: "Which existing behavior must be preserved, and which test proves it?"
evidence: "existing test name + passing run"
- id: q4_rollback
text: "What is the single command or flag that reverses this change?"
evidence: "revert command, not a plan"
- id: q5_owner
text: "Who owns this decision, and where is it recorded?"
evidence: "name + decision file path"

#!/usr/bin/env bash
# review-session.sh <session-dir> <candidate.patch>
set -euo pipefail

SESSION_DIR="${1:?usage: review-session.sh <session-dir> <patch-file>}"
PATCH="${2:?usage: review-session.sh <session-dir> <patch-file>}"
mkdir -p "$SESSION_DIR"

cp "$PATCH" "$SESSION_DIR/candidate.patch"
sha256sum "$SESSION_DIR/candidate.patch" | awk '{print $1}' > "$SESSION_DIR/candidate.sha256"
git rev-parse HEAD > "$SESSION_DIR/base-rev.txt" 2>/dev/null || echo "no-repo" > "$SESSION_DIR/base-rev.txt"
printf '{}\n' > "$SESSION_DIR/answers.json"

cp questions.yml "$SESSION_DIR/questions.yml"
echo "session scaffolded at $SESSION_DIR"

#!/usr/bin/env bash
# gate.sh <session-dir> - blocks until every ladder entry has an answer
set -euo pipefail

SESSION_DIR="${1:?usage: gate.sh <session-dir>}"

jq -e 'all(.[]; (.answer | type) == "string" and (.answer | length) > 0)' \
"$SESSION_DIR/answers.json" > /dev/null \
|| { echo "BLOCKED: unanswered question in the ladder"; exit 1; }

jq -r 'to_entries[] | "\(.key)\t\(.value.answer)\t\(.value.evidence)"' \
"$SESSION_DIR/answers.json" > "$SESSION_DIR/decision.tsv"

echo "PASS: decision record written to $SESSION_DIR/decision.tsv"

The two scripts are deliberately boring, because the value is in the ordering. The hash pins the artifact, the base revision pins the context, and the gate refuses to let a partial conversation look like a completed review.

Where a free model and a free server fit

A candidate patch has to come from somewhere before the ladder can be applied to it. MonkeyCode offers free model access for producing that candidate and a free server option for running the reproduction in a disposable environment rather than on a reviewer's laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. In this workflow the product participates in two narrow steps only: drafting the patch that the session will interrogate, and hosting the clean checkout where questions one and four get verified. Availability, allowances, and terms for both options can change, so confirm the current details on MonkeyCode's own documentation before depending on them for anything scheduled.

Keeping the reproduction off the reviewer's machine matters for a mundane reason. A clean checkout on a disposable host proves the failure exists without local caches, editor plugins, or half-installed dependencies. The session log then records a host and a revision instead of a workstation nobody else can reach.

The decision table the session produces

Limitations and who should not use this

This workflow assumes a repository with a reproducible entry point and at least one automated test that can run headlessly. It is poor value for typo fixes, dependency bumps, and one-line configuration changes, where the ladder costs more than the patch. Teams shipping one candidate patch per week will find the ten-to-fifteen minute session overhead hard to justify, and teams handling regulated data should confirm vendor terms and internal policy before pushing any source to a hosted free service.

The free server option is a disposable environment, and it should never hold production credentials, customer data, or long-lived secrets. The ladder also does not replace a security review, a license scan, or a human who understands the domain the patch touches. Treat the recorded decision file as evidence of process, not as proof of correctness.

Dead ends worth naming

The most expensive dead end is a patch that reproduces the failure only after an uncommitted local edit, because question one then has no clean answer. The second is a read set discovered after the merge, which usually appears as a surprise import in a diff nobody re-reads. Both are cheap to catch at the start of a session and expensive to catch three weeks later during an incident review.

If a team wants to try the pattern without provisioning anything, the free model access and free server option through MonkeyCode are a reasonable place to run one session end to end and see whether the ladder earns its keep.#ai #programming #codequality #devtools #software #coding #development #engineering #inclusive #community
A Senior Asks Five Questions Before Reading Your Generated Patch

The fifteen minutes you budget for a new AI coding setup rarely die from a missing feature, and they almost never die from a slow model. They die from three unnamed waits: a key you hold but have nowhere to put, a command that technically succeeds while producing nothing you can verify, and the absence of any rule telling you when to stop. I have watched those three waits eat entire afternoons, and I stopped trusting my memory about them a long time ago.

So this is not another review of what a tool can do. It is a measurement instrument you can run in your own terminal, plus the one fix that actually moved the needle for me. Call it a friction ledger, and yes, it is less exciting than a benchmark table, which is exactly why it works.

Name the three leaks before you time anything

Placement friction is the cost of getting a credential, an endpoint, and a model identifier into the same place at the same time. Verification friction is the cost of turning a successful HTTP response into something a human can inspect and replay. Recovery friction is everything you pay after a mistake, including the four minutes you spend wondering whether the mistake was yours or the tool's.

None of those show up in a "time to first token" chart, because a token is not a diff and a 200 is not evidence. What do you actually have after the happy path returns? If the answer is "a feeling," you have measured nothing, and the next person on your team will pay the same tax you just paid.

The ledger itself: forty lines of bash

A friction ledger records one row per step, with a monotonic-ish millisecond timestamp at the start and the end of that step. The point is not precision; the point is capturing the shape of the wait while you are still annoyed by it. Save this as friction.sh, then chmod +x friction.sh.

#!/usr/bin/env bash
set -euo pipefail

LEDGER="${LEDGER:-./friction.csv}"
STATE="${STATE:-./.friction.state}"

now_ms() { date +%s%3N; }

init() {
[ -f "$LEDGER" ] || printf 'step,start_ms,end_ms,elapsed_ms,outcome,note\n' >"$LEDGER"
}

start() {
init
local step="${1:?usage: friction.sh start <step>}"
printf '%s %s\n' "$step" "$(now_ms)" >"$STATE"
echo "started: $step"
}

end() {
local outcome="${1:-ok}" note="${2:-}"
local started_step started_at
read -r started_step started_at <"$STATE"
printf '%s,%s,%s,%s,%s,"%s"\n' \
"$started_step" "$started_at" "$(now_ms)" \
"$(( $(now_ms) - started_at ))" "$outcome" "$note" >>"$LEDGER"
echo "$started_step: $(( $(now_ms) - started_at ))ms ($outcome)"
}

summary() {
awk -F, 'NR>1 {t+=$4; n++} END {printf "%d steps, %d ms total, %.0f ms mean\n", n, t, t/n}' "$LEDGER"
}

"$@"

Usage is deliberately boring, because a tool you have to think about is itself friction:

./friction.sh start credential-placement
# ... paste the key, wire the endpoint, pick the model ...
./friction.sh end ok "key in env file, not shell history"

./friction.sh start first-verifiable-output
./friction.sh end blocked "response arrived, no artifact on disk"

./friction.sh summary

One honest caveat about the instrument: it measures your environment and your habits, not the provider's engineering. A slow step might be your proxy, your laptop, or a corporate VPN doing what corporate VPNs do, and the ledger will happily record that as the product's fault. Keep the note column populated and you will at least remember which one it was.

Where a free tier fits into this, without the fantasy math

Free access changes the calculation because the cost of a wrong turn drops to nearly zero, which is precisely when people stop being careful. I have been using MonkeyCode, an open-source project whose operator describes it as offering free model access together with a free server option, and a free allowance he puts at ten million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Because that arrangement is described to me rather than audited by me, I treat it the way I treat any free tier: useful for measuring, dangerous for assuming. Verify the current terms on the project page before you plan a sprint around them, since free allowances move and nobody owes you a grandfather clause.

What the free server option is genuinely good for is the boring middle of the ledger. You want a place to run the first-verifiable-output step that is not your work laptop with three expired tokens in a config directory. What it is not good for is proving that your production topology will behave the same way, because it will not, and pretending otherwise is how teams ship surprises.

The one fix that mattered

Here is the change that cut my own ledger the most, and it had nothing to do with models. I collapsed three separate touchpoints — key, base URL, and model identifier — into one sourced file, so that there was exactly one place where a stale value could hide. Before that, every step in the ledger was quietly paying a config-hunting tax I had been blaming on the network.

# ~/.config/ai/env.sh — sourced, never copy-pasted into a shell history
: "${AI_KEY:?set AI_KEY before sourcing this file}"
: "${AI_MODEL:?set AI_MODEL before sourcing this file}"
export AI_BASE_URL="${AI_BASE_URL:-https://your-endpoint-here}"
export AI_KEY AI_MODEL

Then a preflight that fails loudly and early, before the fifteen-minute clock starts rather than nine minutes into it:

preflight() {
local start elapsed
start=$(date +%s%3N)
for v in AI_BASE_URL AI_KEY AI_MODEL; do
[ -n "${!v:-}" ] || { echo "missing: $v"; return 1; }
done
curl -sS -o /dev/null -w 'http %{http_code}\n' \
-H "Authorization: Bearer $AI_KEY" "$AI_BASE_URL/models" || return 1
elapsed=$(( $(date +%s%3N) - start ))
echo "preflight ok in ${elapsed}ms"
}

Pair it with a step budget, and the ledger stops being a diary and becomes a decision. I use two thresholds: placement must finish inside one minute, and first verifiable output must finish inside five. If either blows past its budget twice in a row on the same step, I stop debugging and change the approach instead of the parameter.

Limitations, and who should skip this entirely

The numbers in that table are placeholders from my own template and not measurements of any specific product, so please do not quote them as benchmarks. A CSV kept by one developer also hides the failures that matter at scale, like the teammate who never tells you they were blocked for three days. And if your code cannot leave your network for compliance reasons, no free server option is a relevant input, no matter how generous the allowance looks.

Who should not use this? Teams that need contractual guarantees, anyone unwilling to re-run the ledger after an environment change, and people who want a benchmark score to paste into a slide. A single measurement run is theater; the ledger earns its keep when it is boring and repeated.

If you want a cheap place to try the whole loop, the MonkeyCode project is open source and its free model access plus free server option is a reasonable sandbox for the first two ledger steps. Bring a stopwatch, trust your own rows over anyone's summary, and tell me which step ate your fifteen minutes — I suspect it was never the model.

# the whole article in two lines
./friction.sh start first-verifiable-output && ./friction.sh end ok "artifact on disk, path recorded"
./friction.sh summary#ai #opensource #productivity #devtools #software #coding #development #engineering #inclusive #community
Onboarding Budgets Leak Before the First Commit

Rubric Graders Need Canaries: Testing the Test That Grades C++ Patches

A pipeline starts scoring C++ patches with a model-backed rubric. Diffs that change printed precision, drop a noexcept guarantee, or leak a file descriptor now receive a 1-10 score before a human opens them. For a few weeks the dashboard looks reasonable, reviewers trust the number, and nobody asks whether the grader still separates good patches from bad ones. Then a prompt edit lands, the upstream model is swapped, and every patch scores 8. Nothing crashes. The grader simply stops discriminating, and the pipeline keeps reporting green.

That failure mode is independent of which model you call. A rubric grader is a program with an untested dependency, and it deserves the treatment the code under it already gets: golden cases, a canary set, a hard gate, and a ledger. The harness below is standard-library Python, a few JSON canary files, and one cron line. It is deliberately boring, because the interesting part is that the grader fails loudly instead of silently.

Layer 0 thinks, layer 1 judges

Split grading into two layers before writing any code. The deterministic layer answers questions with one correct answer: does the patched tree compile, does the golden case exit zero, does stdout match after a fixed locale and precision. The rubric layer answers questions with a range of defensible answers: is the naming consistent with the header it implements, does the commit message describe the actual change, does the new overload respect the documented contract.

Most silent regressions live on the boundary. A grader that is only deterministic cannot see prose contracts. A grader that is only a rubric cannot see a changed std::setprecision. Layer 0 gates the build, layer 1 explains the patch, and each layer needs its own canaries.

A canary here means a labeled example with a fixed expected answer that is committed to the repository. It is not a test of the patch. It is a test of the test.

Step 1 - Freeze golden cases for the deterministic layer

Start with one golden case per documented output contract. Keep the case small enough that a failure points at one behavior, and pin everything that could drift: locale, precision, and the toolchain described in your own build docs.

// golden/report_precision.cpp - contract: ratio printed with 3 decimals
#include <iostream>
#include <iomanip>

int main() {
const double ratio = 2.0 / 3.0;
std::cout << std::setprecision(3) << std::fixed << ratio << std::endl;
return 0;
}

Run it through the same command the eval batch uses, so the grader and the patch pipeline cannot disagree about flags:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target report_precision
./build/report_precision > out.txt
diff -u golden/report_precision.expected out.txt

The diff is the grader. If it passes, the patch is not yet good, but it is at least not obviously broken.

Step 2 - Commit canaries in both directions

A canary set that only contains bad patches rewards a paranoid grader. A canary set that only contains good patches rewards a lenient one. Include both, and store each patch as its own file so the JSON stays readable and diffs stay reviewable.

eval/
canaries/
precision-3-to-2.json
honest-refactor.json
tie-case-review.json
diffs/
precision-3-to-2.patch
honest-refactor.patch
tie-case-review.patch

Each canary declares what it expects and why. The note field is the part humans actually maintain.

{
"id": "precision-3-to-2",
"kind": "rubric",
"direction": "must-fail",
"patch_path": "diffs/precision-3-to-2.patch",
"expected": 3,
"note": "Breaks the output contract in docs/report.md. Any score above 5 means the rubric is not reading the contract."
}

{
"id": "honest-refactor",
"kind": "rubric",
"direction": "should-pass",
"patch_path": "diffs/honest-refactor.patch",
"expected": 8,
"note": "Pure rename plus a comment fix, output byte-identical. Below 6 means the rubric punishes chore diffs."
}

{
"id": "precision-3-to-2-build",
"kind": "build",
"patch_path": "diffs/precision-3-to-2.patch",
"golden_cmd": "./build/report_precision | diff -u golden/report_precision.expected -",
"note": "Must exit non-zero. If this passes, the deterministic layer is not running."
}

Keep the count honest. Twenty labeled canaries drawn from real review comments are worth more than two hundred generated ones, because generated canaries encode the same assumptions as the rubric prompt.

Step 3 - Measure agreement before trusting scores

Agreement is the only cheap health signal a rubric grader has. Exact match is strict; within-one match tolerates the ordinary noise of a 1-10 scale. Both numbers belong in the gate.

#!/usr/bin/env python3
# grade_gate.py - run canaries before trusting rubric scores.
import json, os, pathlib, subprocess, sys

CANARY_DIR = pathlib.Path(os.environ.get('CANARY_DIR', 'eval/canaries'))
LEDGER = pathlib.Path(os.environ.get('GRADER_LEDGER', 'eval/grader_ledger.jsonl'))
RUBRIC_ID = os.environ.get('GRADER_VERSION', 'unversioned')

def load_canaries():
for path in sorted(CANARY_DIR.glob('*.json')):
canary = json.loads(path.read_text())
patch = (CANARY_DIR / canary['patch_path']).resolve()
canary['diff_text'] = patch.read_text()
yield canary

def call_model(prompt):
# Transport layer. Plug in whichever client your runtime provides.
# This function is untrusted by design; that is why canaries exist.
raise NotImplementedError('wire your client here')

def rubric_score(diff_text, rubric):
reply = call_model(rubric + '\n\nDIFF:\n' + diff_text)
return json.loads(reply)

Now the comparison itself, plus the gate. The thresholds are policy defaults, not measured results: a canary set of forty concrete review-derived cases can usually carry a higher bar than one of twelve fuzzy ones. Tune them once, write the number in the repo, and treat any change to it as a reviewable commit.

def check_rubric_canaries(canaries, rubric, min_exact=0.80, min_within_one=0.95):
exact = within_one = 0
misses =
[] for canary in canaries:
if canary.get('kind') != 'rubric':
continue
expected = canary['expected']
got = rubric_score(canary['diff_text'], rubric)['score']
if got == expected:
exact += 1
else:
if abs(got - expected) == 1:
within_one += 1
misses.append({'id': canary['id'], 'expected': expected,
'got': got, 'delta': got - expected})
total = sum(1 for c in canaries if c.get('kind') == 'rubric') or 1
rates = {'exact': exact / total, 'within_one': (exact + within_one) / total}
ok = rates['exact'] >= min_exact and rates['within_one'] >= min_within_one
return ok, rates, misses

Build canaries run the golden command instead of a model, and check the opposite thing: they must fail.

def check_build_canaries(canaries, workdir):
results =
[] for canary in canaries:
if canary.get('kind') != 'build':
continue
subprocess.run(['git', 'apply', canary['patch_path']], cwd=workdir, check=True)
proc = subprocess.run(canary['golden_cmd'], cwd=workdir, shell=True)
subprocess.run(['git', 'restore', '.'], cwd=workdir, check=True)
results.append({'id': canary['id'], 'exit': proc.returncode,
'ok': proc.returncode != 0})
return results

Step 4 - Keep a grader ledger next to the token ledger

A score is only meaningful with the grader revision and the model identity attached. Write one JSON line per gate run, including the token count, so drift and cost are visible in the same file.

def log_run(rates, misses, model_id, tokens):
record = {'grader': RUBRIC_ID, 'model': model_id, 'rates': rates,
'misses': misses, 'tokens': tokens}
with LEDGER.open('a') as fh:
print(json.dumps(record), file=fh)

if __name__ == '__main__':
rubric = pathlib.Path(sys.argv[1]).read_text()
canaries = list(load_canaries())
ok, rates, misses = check_rubric_canaries(canaries, rubric)
build = check_build_canaries(canaries, '.')
log_run(rates, misses, os.environ.get('MODEL_ID', 'unknown'),
int(os.environ.get('TOKENS_USED', '0')))
healthy = ok and all(r['ok'] for r in build)
print(json.dumps({'healthy': healthy, 'rates': rates, 'misses': misses,
'build': build}, indent=2))
sys.exit(0 if healthy else 1)

The exit code is the whole interface. A healthy gate returns 0 and the eval batch proceeds; an unhealthy gate returns 1 and the batch stops before it spends anything on scoring.

Step 5 - Run the gate where nobody has to remember it

A gate that depends on a human typing a command is a gate that will be skipped during the week it matters most. Two things make the scheduled version practical: a machine that stays on, and model access that does not require a per-run decision.

MonkeyCode, described by its maintainers as an open-source project, offers free model access and a free server option, with the free tier described as including 10M tokens. That combination is enough to host a weekly canary run plus the rubric calls it makes, without reserving a dedicated box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# cron: every Monday 06:17, on the free server option
17 6 * * 1 cd /srv/eval && python3 grade_gate.py rubrics/patch_v3.md >> logs/canary.log 2>&1

Read the ledger rather than the estimate. The claim worth trusting is not a predicted token count; it is the tokens field on last week's line, compared with the line before it.

Which layer owns which signal

If a row can be pushed into layer 0, push it. Every signal moved out of the rubric layer removes a source of drift, and the rubric layer keeps only the judgments a compiler cannot express.

Limitations and who should skip this

Canaries are a sample, and agreement is not correctness. If the labels are wrong, the gate certifies a wrong grader with a confident number. Re-label a handful of canaries every few months, especially the ones that never fail.

Rubric scores drift when the model or the prompt changes, and only a scheduled run of the ledger detects that. Single-run scores stay noisy even with a fixed prompt, so treat a one-point swing as a signal to check the ledger, not as a verdict on the patch.

The quota and server claims above are operator descriptions of the offer, not measurements taken for this article. Verify current terms before wiring a paid-critical pipeline to a free tier, and keep the gate able to run against a different client.

Skip this approach if you have fewer than roughly twenty real diffs to label, because reading the patches is then cheaper than grading them. Skip it for safety-critical, licensing, or compliance judgments, which need deterministic evidence and a named reviewer. Skip it if diff text cannot leave your network, unless every layer runs locally. And skip the rubric layer entirely if your grading budget exceeds the cost of the human review it is meant to save.

The harness is small on purpose. Golden cases, canaries in both directions, one agreement gate, one ledger. Start with a single canary that should fail and confirm the gate returns non-zero; if it returns zero, the grader was never watching.#cpp #testing #ai #devtools #software #coding #development #engineering #inclusive #community
Rubric Graders Need Canaries: Testing the Test That Grades C++ Patches

48-Hour Field Notes: Every write() Returned, close() Raised ENOSPC, and My Ledger Ended Mid-Record

The boring loop that turned into a two-day investigation

For most of last week I ran a small harness: take a prompt, ask a model for a patch suggestion, apply it to a scratch checkout, and append one JSON line per attempt to a ledger file. I used MonkeyCode's free model access for the suggestions and its free server option for the scratch host, both of which were available to me during the window described here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not going to quote quotas, limits, or hardware for either of those, because I never measured them and I will not guess. What I can describe precisely is what the ledger did on a small scratch host, and why the first version of it told me a comfortable lie for about nineteen hours.

What the timeline actually looked like

• Hours 0–3. The loop worked. Each attempt appended one line, the file grew, and I watched df -h like a tourist.
• Hours 3–19. I stopped watching. Attempts kept "succeeding," because write() kept returning, and nothing in my code ever asked whether those bytes had reached the disk.
• Hour 19. A later process tried to read the ledger and crashed inside json.loads on the final line. The last record was cut in half, mid-string, with no trailing newline.
• Hours 19–24. I reproduced the failure on purpose by filling a tiny filesystem and rerunning the loop. The error surfaced at close(), not at write().
• Hours 24–48. I rebuilt the writer, added a headroom guard, and added a verifier that can tell me a log is short instead of pretending it is complete.

So which of those five time slices would you have spent debugging the wrong layer? I spent roughly four hours of hour 19 blaming my parser.

Why "No space left on device" arrived at close()

CPython's text-mode files are buffered, and the default buffer size is 8192 bytes. When you call write(), you have only copied bytes into a buffer inside your own process. The real write(2) syscall happens later, during a flush or a close, and that is where ENOSPC appears.

That gives you three very different failure points:

1. write() raises — the record was larger than the buffer, or the buffer was already full.
2. flush() raises — you can name the exact record that never landed.
3. close() raises — the process already believed everything was fine, and the last record on disk may be truncated.

One extra trap is worth knowing. Calling tell() on a text stream flushes the buffer in order to compute an accurate position. If you call tell() after every append, you move the error earlier and lose the clean attribution between "record 61 failed" and "the file was already broken before record 61."

Another nuance cost me time: on ext4, running out of inodes also reports ENOSPC, so "disk full" can mean "no more directory entries." My first version wrote one patch file and one log file per attempt, which is exactly the workload that hits that wall while df -h still shows gigabytes free.

Primary references I checked before writing this: the Python documentation for os.fsync and os.statvfs at docs.python.org, and the Linux manual pages for write(2) and fsync(2).

Step 1: Probe headroom in bytes and in inodes

Run this before the loop, not after it. Both numbers matter, and they fail independently.

mkdir -p /tmp/runs
df -h /tmp
df -i /tmp
findmnt -no SOURCE,FSTYPE,OPTIONS --target /tmp
python3 - <<'PY'
import json, os
st = os.statvfs("/tmp")
print(json.dumps({"free_bytes": st.f_bavail * st.f_frsize, "free_inodes": st.f_favail}))
PY

If your box is a container, treat that output as an optimistic upper bound rather than a promise. statvfs may be describing the underlying filesystem instead of the quota your process actually lives under.

Step 2: A ledger writer that refuses to lie

This is the second version of the writer I used after hour 24. It checks headroom before opening and before each record, flushes and fsyncs per record so a failure names the record, and counts bytes manually instead of calling tell().

#!/usr/bin/env python3
"""ledger.py - append-only JSONL writer for model-run records.

v2 of the writer from a 48-hour field-notes run:
- checks headroom before opening and before each record
- flush + fsync per record, so a failure can name the record
- never calls tell() on a text stream (that flushes and hides attribution)
"""
from __future__ import annotations

import errno
import json
import os
import time

MIN_FREE_BYTES = 64 * 1024 * 1024
MIN_FREE_INODES = 1000

HINTS = {
errno.ENOSPC: "filesystem or quota is full (inodes count too)",
errno.EDQUOT: "container or user quota exceeded",
errno.EFBIG: "file size limit reached",
}

def headroom(path):
st = os.statvfs(path)
return {"free_bytes": st.f_bavail * st.f_frsize, "free_inodes": st.f_favail}

def assert_headroom(path, tag):
h = headroom(path)
if h["free_bytes"] < MIN_FREE_BYTES or h["free_inodes"] < MIN_FREE_INODES:
raise SystemExit(f"[{tag}] refusing to write: {h}")

class Ledger:
def __init__(self, path):
self.path = path
self.directory = os.path.dirname(os.path.abspath(path)) or "."
assert_headroom(self.directory, "open")
self.fh = open(path, "a", encoding="utf-8")
self.seq = 0
self.offset = os.path.getsize(path)

def append(self, **record):
self.seq += 1
payload = {"seq": self.seq, "ts": time.time(), **record}
line = json.dumps(payload, ensure_ascii=False) + '\n'
blob = line.encode("utf-8")
try:
self.fh.write(line)
self.fh.flush()
os.fsync(self.fh.fileno())
except OSError as exc:
hint = HINTS.get(exc.errno, "unexpected")
raise SystemExit(
f"record {self.seq} failed at offset {self.offset}: "
f"errno={exc.errno} ({hint})"
) from exc
self.offset += len(blob)

def close(self):
self.fh.close()

Usage inside the harness is deliberately dull, because a ledger should never be the clever part of your pipeline:

lg = Ledger("/tmp/runs/ledger.jsonl")
for attempt, suggestion in enumerate(suggestions, start=1):
lg.append(attempt=attempt, status="applied", bytes_out=len(suggestion))
lg.close()

Note the trade-off honestly: one fsync per record is not free, and I did not benchmark it for this article. Before you adopt it in a hot path, time a 10,000-append loop with and without the fsync call on your own filesystem and decide from your own numbers.

Step 3: A verifier that tells you the log is short

A writer that fails loudly is only half the defense. The other half is a checker that runs before your replay script trusts the file.

#!/usr/bin/env python3
"""verify_ledger.py - is this JSONL complete, or did it die mid-flush?"""
import json
import sys

def verify(path):
with open(path, "rb") as fh:
data = fh.read()
if data and not data.endswith(b'\n'):
print(f"INCOMPLETE: final record has no newline ({len(data)} bytes)")
return 2
count = 0
for lineno, raw in enumerate(data.splitlines(), 1):
try:
rec = json.loads(raw)
except json.JSONDecodeError as exc:
print(f"line {lineno}: unparsable ({exc})")
return 3
if rec.get("seq") != lineno:
print(f"line {lineno}: seq={rec.get('seq')} out of order")
return 4
count = lineno
print(f"OK: {count} records, contiguous and newline-terminated")
return 0

if __name__ == "__main__":
sys.exit(verify(sys.argv[1]))

The distinct exit codes are the point. A wrapper can branch on 2 (truncated tail, discard the last record), 3 (corruption, stop), or 4 (ordering broken, stop) instead of guessing from a stack trace.

How I reproduced the failure on purpose

Waiting for a real box to fill up is not a test plan. On a machine where loopback mounts are allowed, I made a tiny filesystem and filled it deliberately:

dd if=/dev/zero of=/tmp/small.img bs=1M count=8
mkfs.ext4 -q /tmp/small.img
sudo mkdir -p /mnt/tiny && sudo mount -o loop /tmp/small.img /mnt/tiny
python3 -c 'open("/mnt/tiny/fill", "wb").write(b"x" * (9 * 1024 * 1024))' || true
python3 verify_ledger.py /tmp/runs/ledger.jsonl

If you cannot use loopback mounts, a container with a small writable layer or a deliberately low quota gives you the same classroom. The lesson is identical either way: you want to watch the truncation happen once, on purpose, before it happens to a run you cared about.

The decision table I now use

Limitations, and who should not use this

• os.statvfs may report the host filesystem rather than your container quota, so the guard is an optimistic bound and not a discovery mechanism.
• The only ground truth is an errno from a real write or fsync. The guard reduces damage; it does not tell you what your true ceiling is.
• Per-record fsync costs latency, and I have not benchmarked it here. On network filesystems it can be much worse than on local disk, so batch and verify at the end instead.
• If you already ship logs to a managed collector, or you emit thousands of records per second, skip this entirely. A real log shipper beats a hand-rolled ledger on every axis.
• Do not use any of this to justify a scratch host that is simply too small for your workload. The guard is a smoke detector, not a bigger house.

What I would repeat, and what I would not

I would repeat counting bytes myself instead of calling tell(), writing the verifier on day one rather than day two, and rehearsing the failure on a throwaway filesystem before trusting the happy path. I would not repeat writing one file per attempt on a small box, and I would not repeat watching df -h while ignoring df -i.

Field notes are only useful if they change a habit, so here is mine: the ledger is now allowed to refuse work, and the verifier is allowed to veto a replay. If you have a scratch-host story where the errno and the error message disagreed, I would like to compare notes, especially about which failure your box reports first when the free space runs out.#python #debugging #linux #devtools #software #coding #development #engineering #inclusive #community
48-Hour Field Notes: Every `write()` Returned, `close()` Raised ENOSPC, and My Ledger Ended Mid-Record