Gate your CI on KICS findings with jq: run the scan, count results at or above your severity threshold, exit non-zero if any exist. Hard enforcement, no reliance on --fail-on https://www.valtersit.com/vault/fail-ci-pipeline-on-kics-severity-threshold-using-jq-exit-co-1ca25f/
Search
13 results for “jq”
-
Assert WireMock request counts before teardown with jq against /__admin/requests, fail the pipeline on mismatch. Walkthrough: https://www.valtersit.com/vault/verify-request-journal-assertions-before-tearing-down-mocks-b59360/ #wiremock #verification #ValtersIT
-
Detienen a exfiscal de Chihuahua por caso ligado a muerte de dos agentes de la CIA
César Gustavo Jáuregui Moreno fue detenido en Chihuahua por presuntos delitos de ejercicio ilegal del servicio público y coalición, en una investigación relacionada con un operativo antidrogas en el que murieron dos agentes de la CIA.
Por Martín García | Reportero
El exfiscal general de Chihuahua, César Gustavo Jáuregui Moreno, fue detenido este sábado 17 de octubre por presuntos delitos de ejercicio ilegal del servicio público y coalición, en una investigación relacionada con la participación de agentes de la Agencia Central de Inteligencia de Estados Unidos (CIA) en un operativo antidrogas.
https://youtu.be/jQRhdCeIpks?t=183
La captura se realizó a las 10:20 horas sobre la avenida La Cantera, en la capital del estado, de acuerdo con el Registro Nacional de Detenciones (RND). El exfuncionario quedó a disposición de la Fiscalía General del Estado de Chihuahua (FGE) para continuar con el procedimiento ministerial.
El caso se relaciona con el desmantelamiento de un laboratorio clandestino de drogas sintéticas en la comunidad de El Pinal, municipio de Morelos, donde un operativo realizado en abril derivó en un accidente en el que murieron dos agentes estadounidenses y dos funcionarios mexicanos.
¿Por qué detuvieron al exfiscal de Chihuahua?
La detención deriva de una orden de aprehensión emitida el viernes 16 de octubre por un juez de control, dentro de la causa penal 3055/2026.
De acuerdo con la información difundida sobre la resolución judicial, Jáuregui Moreno enfrenta señalamientos por los presuntos delitos de ejercicio ilegal del servicio público y coalición. Las imputaciones deberán ser examinadas durante el procedimiento judicial, en el que el exfuncionario conserva el derecho a la presunción de inocencia.
El encargado de despacho de la Fiscalía estatal, Gilberto Loya Chávez, informó que la investigación busca esclarecer la probable participación de personal extranjero en el operativo, así como posibles omisiones en la cadena de mando y el incumplimiento de los protocolos aplicables.
La indagatoria no cuestiona únicamente el resultado del despliegue contra el laboratorio clandestino, sino también las circunstancias en las que se realizó y la participación de agentes estadounidenses en territorio mexicano.
¿Quién más tiene orden de aprehensión?
La resolución judicial también contempla a Guillermo Arturo Zuany Portillo, exresponsable de operaciones estratégicas de la Fiscalía estatal.
Según la información disponible sobre el mandamiento judicial, Zuany Portillo es investigado por los presuntos delitos de ejercicio ilegal del servicio público y coalición, además de falsedad ante las autoridades.
Hasta el momento de la información consultada, no había confirmación suficiente para establecer que también hubiera sido detenido. –sn–
Sociedad Noticias¡Conéctate con Sociedad Noticias! Suscríbete a nuestro canal de YouTube y activa las notificaciones, o bien, síguenos en las redes sociales: Facebook, Twitter e Instagram.
También, te invitamos a que te sumes a nuestro canal de información en tiempo real a través de Telegram.
#NoticiasMX #PeriodismoParaTi #PeriodismoParaTiSociedadNoticias #agentesDeLaCIA #CésarGustavoJáureguiMoreno #Cdmx #CIA #coaliciónDeServidoresPúblicos #Destacado #ejercicioIlegalDelServicioPúblico #ElPinal #exfiscalDeChihuahua #FiscalíaDeChihuahua #GuillermoArturoZuanyPortillo #Información #InformaciónMéxico #investigaciónJudicial #laboratorioClandestinoDeDrogas #México #Morena #municipioDeMorelos #noticia #noticias #NoticiasMéxico #NoticiasSociedad #seguridadEnChihuahua #SN #Sociedad #SociedadNoticias #SociedadNoticiasCom #sociedadNoticias #SociedadNoticiasCom -
🛡️ secdb-cli 0.7.0 is out.
It's the open source command-line client for ZEN SecDB: CVE lookup, SSVC, and auditing of packages, SBOMs, dependency manifests, Linux hosts and Docker images.
What's new:
- secdb search: full-text search across CVEs, CWEs, security advisories, NASL plugins and ExploitDB entries. Filter by record type and by date, combine terms with OR, exclude them, search exact phrases. CVEs in the CISA KEV catalog are marked, and every result links to its record.
- --jq: filter the JSON output of any command with a jq expression, without installing jq. Handy in CI images and on Windows.
- Safer terminal output: control characters coming from third-party feed data are stripped before they reach your terminal.
Examples:
secdb search fortinet kev:true --since 30d
secdb search '"remote code execution"' --type cve
secdb audit sbom --file bom.json --fail-on high --jq '.[].advisories[].id'💾 Binaries for Linux, macOS and Windows:
https://github.com/giterlizzi/secdb-cli/releases/tag/v0.7.0#InfoSec #VulnerabilityManagement #OpenSource #Golang #CVE #NTTDATA #ZENSecDB
-
A shared free AI bench rarely dies from an empty allowance. It dies from an unowned next step.
The developer conversation this week mixed two moods. There is fatigue with AI tooling, and there is a sprint to ship open-source work. Both push teams toward the same move: pool a free tier, share it, and hope nobody hogs it. Hope is not a handoff.
MonkeyCode is an open-source coding agent project. The operator states that it offers free model access and a free server option. Treat both as availability, not as a contract.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The failure this SOP prevents
A free bench has no queue and no billing alarm. Two people start long runs, a third retries a failure, and the window closes mid-task. Nobody can say which run burned the last of it.The fix is small and boring. Name three roles, write one card per run, and keep a local token ledger you control. The ledger is yours, so it survives quota changes and product changes.
Three roles, one handoff
The Bench Steward owns the budget line and the ledger file. This person approves a run before it starts and closes it after. The Steward does not write code during a shift.The Runner owns a single run card at a time. One card equals one goal, one model, one output artifact. A second goal needs a second card, even when the first is nearly done.
The Reviewer owns the accept or reject call. This person reads the diff and the card, never the chat history. If the card cannot explain the change, the change is not ready.
Handoffs move in one direction. Runner hands a filled card to Reviewer. Reviewer hands a verdict to Steward. Steward writes the next slot. No role skips forward, and no role edits another role's section.
Artifact one: the token ledger
Put this file atbench/ledger.shand keep the data in plain TSV. It reads the usage block from a saved response, so it never guesses.#!/usr/bin/env bash
# bench/ledger.sh — count tokens per run card, refuse to start over budget
set -euo pipefailBENCH_DIR="${BENCH_DIR:-bench}"
LEDGER="$BENCH_DIR/token-ledger.tsv"
BUDGET_FILE="$BENCH_DIR/budget.txt"[[ -f "$BUDGET_FILE" ]] || { echo "missing $BUDGET_FILE" >&2; exit 2; }
BUDGET=$(tr -dc '0-9' < "$BUDGET_FILE")
[[ -n "$BUDGET" ]] || { echo "budget file has no number" >&2; exit 2; }[[ -f "$LEDGER" ]] || printf 'run_id\towner\tmodel\ttokens\n' > "$LEDGER"
spent() { awk -F'\t' 'NR>1 {s+=$4} END {print s+0}' "$LEDGER"; }
check() {
local now left
now=$(spent)
left=$(( BUDGET - now ))
printf 'budget=%s spent=%s left=%s\n' "$BUDGET" "$now" "$left"
(( left > 0 )) || { echo 'FENCE: no headroom left, stop' >&2; return 1; }
}record() {
local run_id="$1" owner="$2" model="$3" payload="$4" tokens
tokens=$(jq -r '.usage.total_tokens // empty' "$payload")
[[ -n "$tokens" ]] || { echo "no usage block in $payload" >&2; return 1; }
printf '%s\t%s\t%s\t%s\n' "$run_id" "$owner" "$model" "$tokens" >> "$LEDGER"
echo "recorded $tokens tokens for $run_id"
}case "${1:-}" in
check) check ;;
record) shift; record "$@" ;;
*) echo 'usage: ledger.sh check | record <run_id> <owner> <model> <response.json>' >&2; exit 64 ;;
esacThe run loop is three commands. Check headroom, do the work, then record.
mkdir -p bench
cat > bench/budget.txt <<'EOF'
400000
EOF./bench/ledger.sh check
# ... run the agent, save the raw response ...
./bench/ledger.sh record r-2026-10-18-a morgan swe-review response.jsonThe number in
budget.txtis your own planning line, not the provider's quota. Set it below whatever the project currently documents, and re-read that page before each planning cycle. Free quotas move.Artifact two: the handoff card
Paste this block into the team wiki and copy it per run. The card is the artifact a reviewer reads.### Run card r-2026-10-18-a
- Owner: morgan
- Slot: 09:00-11:00 UTC
- Goal (one sentence): add retry jitter to the fetch wrapper
- Model: as listed by the project today
- Files touched: src/fetch.ts, test/fetch.test.ts
- Test command: npm test -- fetch
- Ledger line: recorded, 8,412 tokens
- Open risk: jitter default may break timing test
- Reviewer verdict: pendingA card with a missing test command is a block. A card with no ledger line is a block. A card with two goals is split before review, not after.
Artifact three: a bench smoke test
The free server option deserves the same suspicion as any demo host. Four cheap checks answer the only question that matters: can the team rely on it for the next two hours?#!/usr/bin/env bash
# bench/smoke.sh — cheap checks before a shift on the free server
set -euo pipefail
BASE="${BENCH_BASE_URL:?set BENCH_BASE_URL}"probe() {
local label="$1" want="$2" path="$3" ttl="$4" code
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time "$ttl" "$BASE$path" || true)
if [[ "$code" == "$want" ]]; then
printf '%-22s ok (%s)\n' "$label" "$code"
else
printf '%-22s FAIL (want %s, got %s)\n' "$label" "$want" "$code"
fi
}probe reachable 200 /healthz 5
probe unauth-reject 401 /v1/models 5
probe cold-start 200 /v1/models 30
probe metrics 200 /metrics 5The paths above are placeholders. Swap in the endpoints the project documents, keep the timings, and log the output next to the card. A cold start above thirty seconds is a scheduling fact, not a bug report.
When to use the bench, and when not to
SituationShared free benchPaid keyExploratory refactor, reversible diffYesOptionalWork with a hard deadline todayNoYesAny customer data in the promptNoOnly with a reviewed pathLong batch generationNoYesLimits worth stating plainly
The free allowance and the free server are operator statements, not service guarantees. Quotas, model lists, and uptime can change without notice, so re-read the project page before you plan a sprint. The ledger only counts what you save to disk; a run that skips the record step is invisible to the fence. The smoke test proves reachability, not output quality.Skip this playbook if you are the only person using the bench. A solo user needs a budget note, not three roles. Skip it too if the work touches regulated data, because a shared free host is the wrong place for that conversation.
Start with the ledger. Run it for one week before you invite a second person onto the bench.#ai #opensource #devops #tutorial #software #coding #development #engineering #inclusive #community
A One-Page SOP for a Shared Free AI Bench -
@developers @helpers @admins @_followback #tagspub any idea how to use this?
I got my Friendica instance to follow [email protected] from @pasjrwoctx, but [email protected] has not followed me back, I am assuming my instance never was able to send an actual request, I am not sure if that is an issue on my side or yours, I do like the concept so I am interested in getting this working either with the relays or through the mutual follow, there are a lot of single and small instances outside of the mastadon ethos, that would really benefit from this service;
[ndshotph@semidedi11-ga ~]$ cd ~/social.2ndshot.photos
[ndshotph@semidedi11-ga social.2ndshot.photos]$ curl -i "https://tags.pub/.well-known/webfinger?resource=acct:[email protected]"
HTTP/2 200
content-type: application/jrd+json; charset=utf-8
date: Fri, 16 Oct 2026 16:31:29 GMT
etag: W/"12f-fFny35OE91aSeRkcy+L1d83UP+I"
ratelimit: "steady-get"; r=2396; t=56
ratelimit: "burst-get"; r=996; t=6
ratelimit-policy: "steady-get"; q=2400; w=60; pk=:NzYzNjExMGYwNTRl:
ratelimit-policy: "burst-get"; q=1000; w=10; pk=:NzYzNjExMGYwNTRl:
x-request-id: 9dc18f99-c661-4c2b-9124-e383cafb93ac
content-length: 303{"subject":"acct:[email protected]","aliases":["https://tags.pub/user/_followback"],"links":[{"rel":"self","type":"application/activity+json","href":"https://tags.pub/user/_followback"},{"rel":"http://webfinger.net/rel/profile-page","type":"text/html","href":"https://tags.pub/profile/_followback"}]}[ndshotph@semidedi11-ga social.2ndshot.photos]$ curl -i -H "Accept: application/activity+json" tags.pub/user/_followback
HTTP/2 200
content-type: application/activity+json
date: Fri, 16 Oct 2026 16:31:56 GMT
ratelimit: "steady-get"; r=2371; t=29
ratelimit: "burst-get"; r=986; t=5
ratelimit-policy: "steady-get"; q=2400; w=60; pk=:NzYzNjExMGYwNTRl:
ratelimit-policy: "burst-get"; q=1000; w=10; pk=:NzYzNjExMGYwNTRl:
x-request-id: 074f59bc-3359-474c-b89e-fa543483e4eb
content-length: 1728{
"@context": [
"https://www.w3.org/ns/activitystreams",
"https://w3id.org/security/v1",
"https://purl.archive.org/socialweb/webfinger",
"https://purl.archive.org/miscellany"
],
"id": "https://tags.pub/user/_followback",
"type": "Service",
"inbox": "https://tags.pub/user/_followback/inbox",
"webfinger": "[email protected]",
"publicKey": {
"id": "https://tags.pub/user/_followback/publickey",
"type": "CryptographicKey",
"owner": "https://tags.pub/user/_followback",
"publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA9dUt92enb1zUV2ekwZuD\niCcVh32bwJLsIaHZ6i7rpwmFgdTvSifRBLwRTyvt7TrW/NsEN+4Ww+fZ0qOOPmAk\nYSboAAn94b6LhM3eMT3bE89iGtQGByGg49RS6JLgizoDhxqoDA6969EHp/z3q0dz\nmKcZbVc3juFqek60R/bLP5OgePjqQ94VT5kjZ/yWaNubbOeeAxFflF0SCv+WyswJ\nFgq7t+PtFDn/5YHPfM1Tudg9D0LfSuA6z8wNIx9UpnocwVgGotoONN2SBAbiY+ci\np8Gqm0no4uL78WbnzhnklhO6KFnxjF1WFtA+MSck6Si3KEdrmQjmE0HJ9EgSOwtS\njwIDAQAB\n-----END PUBLIC KEY-----\n",
"to": "as:Public"
},
"endpoints": {
"sharedInbox": "https://tags.pub/shared/inbox"
},
"followers": "https://tags.pub/user/_followback/followers",
"following": "https://tags.pub/user/_followback/following",
"liked": "https://tags.pub/user/_followback/liked",
"manuallyApprovesFollowers": false,
"name": "FollowBackBot",
"outbox": "https://tags.pub/user/_followback/outbox",
"preferredUsername": "_followback",
"summary": "\nFollow me and I will follow you back. Your public posts will\nthen be boosted through tags.pub. Unfollow me,\nand the boosts will stop.\n",
"to": "as:Public",
"url": {
"type": "Link",
"href": "https://tags.pub/profile/_followback",
"mediaType": "text/html"
}
}[ndshotph@semidedi11-ga social.2ndshot.photos]$ curl -s tags.pub/profile/_followback | grep -i 'application/activity+json' -n || true
37: link.type = 'application/activity+json'
[ndshotph@semidedi11-ga social.2ndshot.photos]$ curl -s -H "Accept: application/activity+json" tags.pub/user/_followback/foll… | jq .
-bash: jq: command not found
(23) Failed writing body
[ndshotph@semidedi11-ga social.2ndshot.photos]$ bin/console relay add tags.pub/user/_____relay_____/…
tags.pub/user/_____relay_____/… wasn't found
[ndshotph@semidedi11-ga social.2ndshot.photos]$ bin/console relay add tags.pub/user/_____relay_____
tags.pub/user/_____relay_____ is no relay actor
[ndshotph@semidedi11-ga social.2ndshot.photos]$ bin/console contact add 3 tags.pub/user/_followback
[Error] User not found
[ndshotph@semidedi11-ga social.2ndshot.photos]$ bin/console contact add 3 [email protected]
[Error] User not found
[ndshotph@semidedi11-ga social.2ndshot.photos]$ bin/console contact add 3 tags.pub/@_followback
[Error] User not found
[ndshotph@semidedi11-ga social.2ndshot.photos]$ -
Why does a truncated model answer look exactly like a finished one when the only thing you check is the HTTP status code?
That question cost me most of a weekend. I had wired a small summarizer to a streaming endpoint, watched it print a paragraph, and shipped it. Two days of field notes later, I had learned that "HTTP 200" tells you nothing about whether the answer actually completed, and that my parser was happily reporting success on streams that never sent a terminator.
Here is what I tried, what broke, and the probe harness I now run before trusting any streaming endpoint.
The symptom: a paragraph that stopped mid-thought
In my run, the response body ended cleanly at the socket level, the status line said 200, and my client wrote whatever it had buffered. Nothing raised. The output simply stopped mid-sentence, which readers interpret as a model being terse rather than a transport being over. Because I was printing tokens as they arrived, I never compared what I received against what the protocol promised me.That is the trap with streaming: the happy path and the failure path share the same first N bytes. If you only assert on status codes and on the existence of output, both paths pass.
What I instrumented instead of what I assumed
I stopped eyeballing output and wrote a probe that records protocol facts rather than vibes. It reads the response line by line, assembles SSE frames, and records four things per request: time to first byte, whether afinish_reasonappeared in any JSON payload, whether the[DONE]sentinel arrived, and how the socket ended.# stream_probe.py — stdlib only, so it runs anywhere
import json, sys, time, urllib.requestdef parse_frame(lines):
"""SSE frames are newline-delimited; data may span several lines."""
payload =
[] for line in lines:
if line.startswith(":"):
continue # comment / keep-alive
if line.startswith("data:"):
payload.append(line[5:].lstrip(" "))
if not payload:
return None, False
joined = "\n".join(payload).strip()
if joined == "[DONE]":
return None, True
try:
return json.loads(joined), False
except json.JSONDecodeError:
return {"_unparsed": joined}, Falsedef find_finish_reason(obj):
if isinstance(obj, dict):
if obj.get("finish_reason"):
return obj["finish_reason"]
for v in obj.values():
hit = find_finish_reason(v)
if hit:
return hit
elif isinstance(obj, list):
for v in obj:
hit = find_finish_reason(v)
if hit:
return hit
return Nonedef probe(url, payload, headers, read_timeout):
rec = {"ttfb_s": None, "frames": 0, "finish_reason": None,
"done_sentinel": False, "unparsed_frames": 0, "error": None}
req = urllib.request.Request(
url, data=json.dumps(payload).encode(), headers=headers, method="POST")
started = time.monotonic()
frame_lines =
[] try:
with urllib.request.urlopen(req, timeout=read_timeout) as resp:
for raw in resp:
if rec["ttfb_s"] is None:
rec["ttfb_s"] = round(time.monotonic() - started, 3)
line = raw.decode("utf-8", "replace").rstrip("\r\n")
if line:
frame_lines.append(line)
continue
obj, done = parse_frame(frame_lines)
frame_lines =
[] if done:
rec["done_sentinel"] = True
if obj is not None:
rec["frames"] += 1
if "_unparsed" in obj:
rec["unparsed_frames"] += 1
else:
rec["finish_reason"] = rec["finish_reason"] or find_finish_reason(obj)
except Exception as exc: # read timeouts land here on purpose
rec["error"] = f"{type(exc).__name__}: {exc}"
rec["elapsed_s"] = round(time.monotonic() - started, 3)
return recif __name__ == "__main__":
url = sys.argv
[1] body = json.loads(sys.argv[2])
hdrs = json.loads(sys.argv[3])
print(json.dumps(probe(url, body, hdrs, read_timeout=20.0), indent=2))Then a soak loop, because one clean request proves almost nothing:
for i in $(seq 1 20); do
python stream_probe.py "$URL" "$BODY" "$HEADERS" | jq -c '{ttfb_s,frames,finish_reason,done_sentinel,unparsed_frames,error}'
done | tee soak.jsonl
jq -s 'group_by(.finish_reason) | map({reason: .[0].finish_reason, n: length})' soak.jsonlThe four endings the harness separates
Once I classified results instead of staring at them, the ambiguity collapsed into a small set of observable outcomes:- Clean finish. A
finish_reasonarrived and[DONE]followed it, so the answer is complete and my parser can trust the buffer. - Reason without sentinel. The payload carried a
finish_reasonbut the stream closed before[DONE], which suggests a proxy that buffers or cuts the tail frame. - Sentinel without reason.
[DONE]arrived early, which usually means the server truncated generation server-side and reported nothing useful about why. - Neither. The socket closed with a partial sentence and no metadata, which is the case that burned me, and the only honest response is to mark the answer as incomplete and retry or surface it to the caller.
Only case one deserves a green check. Everything else should set an
incompleteflag on the record you store, because a truncated summary that silently enters your database is worse than a visible error.Where a free server and free models actually fit
I run this probe from a throwaway box precisely because I want the transport to be the boring part. MonkeyCode offers free model access and a free server option, which makes it a reasonable place to park a short soak loop like the one above while you are still deciding whether an endpoint deserves production traffic.Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The important discipline is not to treat "free" as "verified". Whatever endpoint you point the probe at, read that provider's current limits and terms yourself on the day you run it, because quotas, model availability, and server lifetimes change and any number I copied here would be stale by the time you read it.
A decision table I now keep in the repo
Signal from the probeWhat I do nextfinish_reasonplus[DONE]on 20/20 runsWire it into the pipeline and keep the probe as a canaryOccasional missing[DONE], reason presentParse per frame, tolerate the tail, log a counterFrequent missingfinish_reasonTreat every response as untrusted text, never as finalRead timeouts clustered at a fixed secondSuspect an idle proxy timeout, not model slownessunparsed_framesabove zeroFix framing first; never debug content on a broken parserWhat I would repeat and what I would skip
I would repeat the soak loop and the classification, because those two together turned a spooky intermittent bug into a checklist. I would skip the hours I spent reading model output for clues about whether the generation was "good", since the transport was broken the whole time and no amount of reading prose would have revealed that.I would also repeat running the probe from a machine I do not care about. Wrapping the read in a hard timeout and letting exceptions land in the record, rather than crashing the loop, is what made the soak results readable.
Limitations and who should not bother
This workflow is for streaming text responses over SSE; if your SDK already exposes a completion flag you trust, a hand-rolled probe is mostly redundant. It also cannot tell you why a provider cut a stream, only that the frames disagree with each other, so pair it with the provider's own logs when you escalate. And if your workload is batch and non-streaming, a single content-length check will beat all of this.If you keep one habit from these field notes, make it this: assert on the terminator, not on the status code. A stream that never says goodbye is not a finished answer, no matter how confident the first paragraph looks.#python #api #debugging #testing #software #coding #development #engineering #inclusive #community
48-Hour Field Notes: My Stream Ended Without [DONE] and the Client Called It Success - Clean finish. A
-
I opened two terminals at 08:20 and closed them at 19:40.
One terminal held a free server box. The other held an agent writing patches.
By lunch I had three diffs and no idea which run produced which.This is a letter to that version of me. It is also a workflow you can copy.
It uses free model access as a second reader, and a free server as the test runner.
Both are operator-supplied availability claims, not performance promises.What the day actually cost
I lost roughly three hours re-running the loop. I could not match diffs to runs.
I lost another two hours debugging a patch a reviewer model had blessed.
The rest went to a fixture directory that changed hash between runs.None of those failures were the model's fault. All three were missing counters.
Mistake 1: I let the harness count attempts, not me
The agent retried. The wrapper retried. The shell loop retried again.
Three layers of retry, one log file, zero timestamps. That is not observability.Fix: write a ledger row before the first model call, not after the last.
The ledger is a TSV file. It is boring and it saved the next day.#!/usr/bin/env bash
# Adapted example. Not run as-is; substitute your own paths and commands.
set -euo pipefailRUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
ART="./runs/$RUN_ID"
mkdir -p "$ART"
LEDGER="./ledger.tsv"log() { printf '%s\t%s\t%s\t%s\n' "$RUN_ID" "$1" "$2" "$3" >> "$LEDGER"; }
FIXTURE_HASH="$(find fixtures -type f -print0 | sort -z \
| xargs -0 sha256sum | sha256sum | cut -d' ' -f1)"
log fixture "$FIXTURE_HASH" "frozen-input"
log attempt "1" "start"The rule I now follow: no ledger row, no model call.
A run without a row is a rumour, even if the diff looks correct.Mistake 2: I treated reviewer agreement as a test
I asked a model to check a patch. It said the patch looked correct.
I merged it. A unit test then failed on an off-by-one in the range check.A reviewer model produces an opinion. Your test suite produces an exit code.
Only one of those two things can block a merge in your CI.Fix: the reviewer sees the diff and the failing output. Nothing else.
Feed it the smallest context that still explains the defect.# Adapted example: reviewer input is diff + failing test output only.
git diff --no-color > "$ART/patch.diff"
pytest -q 2>&1 | tail -n 40 > "$ART/test_tail.txt" || true# Build the reviewer payload from those two files, not from the repo.
printf '%s\n' 'Review this diff against this test failure.' \
'Return JSON: {"verdict":"agree|disagree","reason":"..."}' \
'--- DIFF ---' > "$ART/review_prompt.txt"
cat "$ART/patch.diff" >> "$ART/review_prompt.txt"
printf '%s\n' '--- FAILURE ---' >> "$ART/review_prompt.txt"
cat "$ART/test_tail.txt" >> "$ART/review_prompt.txt"Then compare two things: the reviewer verdict and the test exit code.
If they disagree, stop. A human reads the run before anything else happens.Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Mistake 3: I ran the writer and the reviewer in one workspace
The writer agent wrote scratch files. The reviewer read that polluted directory.
The next run's fixture hash no longer matched the ledger, so the diff was unprovable.Fix: separate directories, one hash gate, and a write-path allowlist.
Run the gate before any shell command executes. Not after.# 1) Plan must be small, typed, and declare its write paths.
jq -e '
(.steps | length) > 0 and
(.steps | length) <= 5 and
has("write_paths") and
all(.steps[]; (.cmd | type) == "string")
' plan.json > /dev/null# 2) Write paths must stay inside the allowed trees.
jq -r '.write_paths[]' plan.json | while read -r p; do
case "$p" in
./src/*|./tests/*) ;;
*) echo "refusing path: $p" >&2; exit 3 ;;
esac
done# 3) Fixture pack must be unchanged since the ledger row was written.
NOW="$(find fixtures -type f -print0 | sort -z \
| xargs -0 sha256sum | sha256sum | cut -d' ' -f1)"
[ "$NOW" = "$FIXTURE_HASH" ] || { echo "fixture drift" >&2; exit 4; }The loop that survived the week
- Freeze the fixture pack and log its hash before anything else.
- Write the plan as JSON. Validate it with
jqand the allowlist gate. - Run the test suite on the free server option, capture exit code and tail.
- Send only the diff and the failure tail to the free model access pass.
- Compare verdict with exit code. Disagreement halts the run.
- Append one ledger row per step, including wall-clock seconds.
Six steps produce a receipt you can replay on Friday. That is the whole point.
When free capacity is the wrong tool
SituationLocal onlyFree serverPaid, fixed SLASingle-file fix, tests under 30sGood fitOptionalOverkillNeeds system packages or servicesSlowGood fitGood fitUnattended multi-hour loopRiskyRiskyGood fitAnything touching credentialsNeverNeverIsolated runner onlySecond-reader pass on a diffGood fitGood fitOptional
Free access is availability, not a throughput guarantee. I plan short bursts, not deadlines.
I also assume nothing about persistence. Artifacts come back to my machine immediately.
Three rows are judgement calls. Two rows are hard limits, not preferences.Limitations and who should skip this
Free model access and the free server option are operator-supplied claims. Verify terms yourself.
I did not measure throughput, and I would not schedule a release on it.Skip this workflow if your tests need production credentials or regulated data.
Skip it if you need contractual uptime or data residency guarantees.
Skip it if a human reviewer cannot read the ledger before merge.What I would tell 08:20-me
Write the counter before you write the prompt. Write the hash before the diff.
When the reviewer and the test disagree, believe the test and call a human.If you want to try this, run the six-step loop once against free model access and the free server option. Keep the ledger even if you stop using both tomorrow.#ai #testing #devops #productivity #software #coding #development #engineering #inclusive #community
Letter to 08:20-Me: Stop Auditing the Agent, Audit the Counters -
Two days ago I started a batch of prompt evaluations, closed the laptop lid, and assumed the free server would keep chewing through the queue while I slept. It did not. When I opened the terminal the next morning, the worker was gone and the only output was a JSONL file that stopped in the middle of a record. Why do we keep trusting long-running processes we never instrumented?
This is a field note about the 48 hours that followed: the naive loop I wrote, the four ways it lied to me, and the small ledger I now put in front of every batch that outlives a single shell session.
What I built first, and why it felt fine
The first version was about thirty lines long and it did exactly what the happy path needed. I pointed a client at MonkeyCode's free model access, ran the script on the free server option, and appended one JSON line per result. Nothing about that loop looked wrong at the time.Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The caps and version strings in the code below are placeholders from my own config, not benchmark results or vendor limits; substitute your own numbers.
# naive_batch.py - the version that lost a night
import json, pathlib, urllib.requestitems = [line.strip() for line in open('prompts.txt') if line.strip()]
out = pathlib.Path('results.jsonl')for i, prompt in enumerate(items):
body = json.dumps({'prompt': prompt}).encode()
req = urllib.request.Request(ENDPOINT, data=body, headers={'Content-Type': 'application/json'})
with urllib.request.urlopen(req, timeout=60) as resp:
text = json.loads(resp.read())['text']
with out.open('a') as f: # append, no fsync, no key
f.write(json.dumps({'i': i, 'prompt': prompt, 'text': text}) + '\n')Read it again and count the assumptions. The index
iis the only identity an item has, the open file handle is the only progress record, and the loop has no idea what it already finished. If the machine disappears, everything the process knew disappears with it.What broke, in the order I noticed it
- Progress lived in memory. The
enumeratecounter was the resume state, so a fresh process always started at zero and re-sent everything. - Retries had no identity. When one call timed out and I re-ran the loop, the same prompt produced a second row with no way to tell which row was the retry.
- Writes were not atomic. A worker killed mid-
writeleft a line that could not be parsed, and my summary script crashed on the last record instead of skipping it. - I could not separate queued from finished. A prompt that never started and a prompt that returned an empty answer looked identical in the file.
Which of those is a model problem? None of them. Every failure came from my bookkeeping, and that was the uncomfortable part of the 48 hours.
The fix: resume by content hash, not by hope
The new version gives each work item a key derived from the thing that actually determines the answer: the normalized prompt, the prompt version, and a model alias I control. Then it reads the ledger before it sends anything.The ledger module
# ledger_batch.py - resume by content hash, not by hope
import hashlib, json, os, pathlibLEDGER = pathlib.Path('ledger.jsonl')
PROMPT_VERSION = 'eval-v3'
MODEL_ALIAS = 'free-default' # an alias you own, not a vendor model id
MAX_ITEMS_THIS_RUN = 40 # placeholder: pick from your own time budget
MAX_ATTEMPTS = 3def normalize(prompt: str) -> str:
return ' '.join(prompt.split()).strip().lower()def key_for(prompt: str) -> str:
raw = '\x1f'.join([PROMPT_VERSION, MODEL_ALIAS, normalize(prompt)])
return hashlib.sha256(raw.encode('utf-8')).hexdigest()def load_state() -> tuple[set[str], dict[str, int]]:
ok, attempts = set(), {}
if not LEDGER.exists():
return ok, attempts
for line in LEDGER.read_text(encoding='utf-8').splitlines():
try:
rec = json.loads(line)
k = rec['key']
except (json.JSONDecodeError, KeyError):
continue # a half-written tail line is expected after a hard kill
if rec.get('status') == 'ok':
ok.add(k)
else:
attempts[k] = attempts.get(k, 0) + 1
return ok, attemptsdef append(record: dict) -> None:
line = json.dumps(record, sort_keys=True, ensure_ascii=False)
with LEDGER.open('a', encoding='utf-8') as f:
f.write(line + '\n')
f.flush()
os.fsync(f.fileno()) # the point is to survive the worker vanishingdef call_model(prompt: str) -> str:
... # your client, your endpoint, your retry policyThe loop that uses it
def run(prompts: list[str]) -> None:
ok, attempts = load_state()
todo = [p for p in prompts if key_for(p) not in ok]
todo = [p for p in todo if attempts.get(key_for(p), 0) < MAX_ATTEMPTS]
print(f'{len(ok)} ok, {len(todo)} remaining, cap {MAX_ITEMS_THIS_RUN}')for prompt in todo[:MAX_ITEMS_THIS_RUN]:
k = key_for(prompt)
try:
text = call_model(prompt)
except Exception as exc:
append({'key': k, 'status': 'error', 'error': type(exc).__name__})
continue
append({'key': k, 'status': 'ok', 'prompt': prompt, 'text': text,
'prompt_version': PROMPT_VERSION, 'model_alias': MODEL_ALIAS})Two properties matter more than the code style. First, re-running the script is cheap, because finished keys are filtered before any network call happens. Second, changing
PROMPT_VERSIONorMODEL_ALIASmakes every old key stale on purpose, so a tweaked prompt never inherits yesterday's answers.Inspecting the ledger without a database
wc -l ledger.jsonl
jq -r 'select(.status=="ok") | .key' ledger.jsonl | sort | uniq -d # should print nothing
jq -r 'select(.status=="error") | .error' ledger.jsonl | sort | uniq -c | sort -rn | headThe duplicate check is the one I run before I trust any summary. If
uniq -dprints keys, some code path is appending twice, and that is a bug in the writer rather than a quirk of the model.A decision table for when this is worth the ceremony
SituationAdd a ledger?ReasonTen calls, you are watching the terminalNoThe bookkeeping costs more attention than the runA batch that outlives your SSH sessionYesResume without re-sending finished workPrompt or model swapped mid-experimentYes, and version the keyOtherwise stale results look freshSeveral workers on the same fileNot this designAn append-only file has no lease or lockResults that must never duplicate for billingNot enoughYou need dedupe at the sink, not only a hash
Does the table feel too cautious? Good. The ledger is cheap insurance for batches that span hours, and pure overhead for scripts that finish before your coffee does.What I would repeat, and what I would change
- Repeat: the per-run cap. It turned an unbounded night into a predictable slice I could inspect in the morning.
- Repeat:
prompt_versionandmodel_aliasinside the key, because they are the two edits that silently invalidate everything. - Repeat:
fsyncon append. It is one line, and it changed a corrupted tail into a recoverable one. - Change: move the ledger off the free server's disk. If the instance is replaced, a local ledger goes with it, so write to storage you control.
- Change: log the request start time, so I can distinguish a slow item from one that never started.
- Change: keep the prompt text in the ledger row. Reconstructing inputs from an index was a mistake I will not repeat.
Limits, and who should not do this
This pattern is not exactly-once delivery, and I want to be blunt about that. A crash between the model response and thefsyncstill costs one duplicate call on resume, and nothing here gives you ordering across multiple workers. Theuniq -dcheck finds duplicates after the fact; it does not prevent them.Free tiers are also not a contract. The free model access and free server option I used were fine for this batch, but availability, capacity, and terms can change, so treat both as replaceable and keep the ledger portable. Anyone reconciling money per request, running parallel consumers, or needing sub-second scheduling should use a real queue with leases and idempotency keys instead of a text file.
Who is this for, then? Developers running evaluation batches, backfills, or slow report jobs on infrastructure they do not fully control, and who would rather resume than restart. If that is you, run ten items first, watch the ledger grow, and only then let the batch run overnight.#python #devops #tutorial #ai #software #coding #development #engineering #inclusive #community
48-Hour Field Notes: The Batch Worker Vanished and My Re-Run Redid Every Item - Progress lived in memory. The
-
The swap everyone makes
You have a paid endpoint and a bill you dislike. A free endpoint appears. You change one line, and the invoice drops to zero. So why does the work feel slower two weeks later?That is the myth I keep meeting. Free model access moves the bill. It rarely deletes it.
Free model access and a free server option are the two availability claims behind this post. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I will not quote quotas, hardware, or latency numbers I have not verified. You measure your own.
Myth 1: Free means the cost is zero
The invoice is zero. The cost moved somewhere else.- Attempt cost. Every retry is another request.
- Wall-clock cost. A queue you wait in is still time spent.
- Human cost. Someone re-runs the job at 2am.
- Opportunity cost. You stop tuning because "it's free anyway."
Add those four lines and the endpoint stops looking free. It is cheap in dollars and expensive in attempts. Do you know your attempts per successful task today?
Myth 2: It is the same model, only cheaper
This one bites hardest. The same family name is not the same serving path.Rate limits and timeouts can differ per deployment. So can context handling and truncation behavior. A familiar model name tells you nothing about the endpoint behind it.
Do not assume parity. Probe it. Send one fixed prompt many times. Record latency, output length, and whether the answer is usable.
If output length shrinks across a long run, that is a serving difference. It is not a bad afternoon. Corrected model: treat a free endpoint as a different deployment, not a discount on the same one.
Myth 3: It is free, so fan it out
Fan-out multiplies attempts, not just traffic. Ten parallel CI jobs become one queue.Queues become timeouts. Timeouts become retries. Retries become duplicate work. Now the free pipeline is slower than the paid one it replaced.
My rule is blunt. Fan-out is real when attempts per success stays near one. Above that, your parallelism is fake. You are only moving load around.
Myth 4: A free server means I can skip setup scripts
No. It means setup has to be scripted, because yesterday's state may not exist.If your runbook lives in your head, a fresh box will expose it. That is not a free-tier problem. That is a reproducibility problem you were hiding.
Corrected model: free compute rewards scripts and punishes memory.
Build a 40-line attempt ledger
Here is the artifact. It wraps any OpenAI-compatible endpoint. It logs every attempt as JSONL, so you can count real work units.Save it as
attempt-ledger.sh. SetENDPOINT,MODEL, andTASKSbefore running. Three attempts per task is the retry ceiling.#!/usr/bin/env bash
# attempt-ledger.sh - log every attempt, not just the happy path.
set -uo pipefailENDPOINT="${ENDPOINT:?set ENDPOINT to your base URL}"
MODEL="${MODEL:?set MODEL}"
TASKS="${TASKS:-20}"
LEDGER="${LEDGER:-attempts.jsonl}": > "$LEDGER"
for i in $(seq 1 "$TASKS"); do
attempt=0
ok=0
while [ "$attempt" -lt 3 ] && [ "$ok" -eq 0 ]; do
attempt=$((attempt + 1))
start=$(date +%s%3N)
code=$(curl -s -o /tmp/ledger_out.json -w '%{http_code}' \
-X POST "$ENDPOINT/chat/completions" \
-H 'Content-Type: application/json' \
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with the single word: ok\"}]}")
end=$(date +%s%3N)
[ "$code" = "200" ] && ok=1
printf '{"task":%d,"attempt":%d,"http":%s,"ms":%d,"ok":%d}\n' \
"$i" "$attempt" "$code" "$((end - start))" "$ok" >> "$LEDGER"
done
doneecho "wrote $LEDGER"
Now summarize it. This jq program prints the numbers worth arguing about.
jq -s '
def p(q): (map(.ms) | sort) as $s | $s[((($s | length) - 1) * q) | floor];
([.[] | select(.ok == 1)] | length) as $ok |
{
logical_tasks: ([.[].task] | unique | length),
total_attempts: length,
successes: $ok,
attempts_per_success: (if $ok == 0 then null else (length / $ok) end),
p50_ms: p(0.50),
p95_ms: p(0.95)
}
' attempts.jsonlWhat follows is an illustrative shape only. These are not measured numbers from any product. Your run will differ.
{"logical_tasks":20,"total_attempts":27,"successes":20,"attempts_per_success":1.35,"p50_ms":1180,"p95_ms":4300}
How to read those numbers
- attempts_per_success = 1.0? The endpoint behaves like a paid one for this workload.
- 1.0 to 1.5? Acceptable for offline batch work.
- Above 1.5? You are paying in time. Fix payload size or concurrency.
- successes = 0? Stop. That is a configuration bug, not a free-tier bug.
A decision table keeps the switch honest.
WorkloadRetry budgetGate before switchingVerdictInteractive chat, human waiting1, manualp95 under your patienceFree access is fineOffline batch, idempotent tasks3attempts/success at or under 1.5Free access is fineCI on every push, wide fan-out0-1queue time unknown, so measureMeasure firstAnything paged at 3am0 without alertingfailure alerts existDo not run free-onlyLong stateful jobsn/astate survives a restart?NoWhat the ledger cannot see
Honesty matters more than a clean table.- It does not measure answer quality. Read some outputs yourself.
- It does not measure your own re-run time. Add a column if you want it.
- It does not prove persistence. A server that lived yesterday may not live tomorrow.
- It does not tell you where your data goes. Ask before sending sensitive prompts.
Who should skip this approach
- Teams with data-residency constraints they cannot verify.
- Products with hard latency budgets and no fallback path.
- Jobs that keep state in memory or on local disk between runs.
- Teams that will not read a ledger file. The number is the whole point.
The corrected mental model is simple. Free model access trades dollars for attempts, queues, and attention. Free servers trade permanence for setup discipline. Neither is a discount on a system you refused to measure. If you run this ledger against MonkeyCode's free model access, tell me your attempts-per-success. That is the only comparison worth making.#ai #devops #testing #productivity #software #coding #development #engineering #inclusive #community
Myth FAQ: Free Model Access Moves the Bill, It Doesn't Delete It -
Detect Silent Model Drift With a Stdlib-Only Golden Set Harness
A hosted model can change between two of your deploys, and nothing in your lockfile moves. Your unit tests still pass, your container still builds, and your users still get slightly different JSON than they got last Tuesday. The practical fix is not a bigger test suite; it is a frozen golden set, a deterministic scorer, and an append-only ledger that fails the run when the pass rate drops below a floor.This article builds exactly that, using only the Python standard library. It runs on a free-tier host, it costs one file of input data, and it answers a single narrow question: did the contract my product depends on stop holding?
Why ordinary tests miss model drift
Application tests assert on code you control. A model response is an input from outside your repository, and it arrives as free text that happens to be parseable.When a provider adjusts decoding defaults, swaps capacity, or routes you to a different checkpoint, the shape of the output can shift while the API status code stays
200. Nothing throws. The failure surfaces later, as a support message about a field that used to be filled in.A golden set inverts that. Instead of asserting on the current behavior, it asserts on a small number of hand-checked examples that were correct on a specific day, and it keeps the receipts.
Step 1: Freeze a small golden set as JSONL
The input file is plain JSONL, one contract per line. Keep it small on purpose: 20 to 40 cases is enough to catch a formatting or label flip, and it fits inside a free-tier request budget.{"id":"invoice_date","input":"Invoice paid 2026-03-04, net 30 days.","expect":{"label":"invoice","fields":{"due_date":"2026-04-03"}}}
{"id":"refund_request","input":"I want my money back for order 8812, it never arrived.","expect":{"label":"refund","fields":{"order_id":"8812"}}}
{"id":"no_action","input":"Thanks, the docs answered my question.","expect":{"label":"none","fields":{}}}Every line has three parts: an
idyou will grep for in the ledger, the rawinputyou send, and theexpectobject you compare against. Do not paste generated examples into this file. If you did not read the output and agree with it, it does not belong in the baseline.Step 2: Write a transport adapter that assumes nothing
The runner below assumes an OpenAI-style/chat/completionsrequest body. That assumption is isolated in one function so you can replace it when your endpoint differs.#!/usr/bin/env python3
"""golden_run.py - regression run against a chat endpoint. Stdlib only."""
import json, os, sys, time, hashlib, pathlib, urllib.requestBASE = os.environ["MODEL_BASE_URL"].rstrip("/")
KEY = os.environ["MODEL_API_KEY"]
MODEL = os.environ["MODEL_ID"]
GOLDEN = pathlib.Path(os.environ.get("GOLDEN", "golden.jsonl"))
LEDGER = pathlib.Path(os.environ.get("LEDGER", "ledger.jsonl"))SYSTEM = (
"You are a strict extractor. Return ONLY minified JSON shaped as "
'{"label": string, "fields": object}. Omit absent fields. '
"No prose, no markdown fences."
)def call(text, timeout=30):
body = json.dumps({
"model": MODEL,
"temperature": 0,
"messages": [{"role": "system", "content": SYSTEM},
{"role": "user", "content": text}],
}).encode()
req = urllib.request.Request(
f"{BASE}/chat/completions", data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {KEY}"},
)
with urllib.request.urlopen(req, timeout=timeout) as r:
payload = json.loads(r.read())
return payload["choices"][0]["message"]["content"]Keep
temperatureat zero when the endpoint supports it. If the parameter is rejected or ignored, note that in your README rather than deleting it, because a later comparison depends on knowing what you actually sent.Step 3: Score the contract, not the prose
Scoring is exact after light normalization: trim, lowercase, collapse whitespace. That is deliberately unforgiving, because the goal is to detect change, not to grade writing.def norm(v):
return " ".join(v.strip().lower().split()) if isinstance(v, str) else vdef score(case, raw):
try:
got = json.loads(raw)
except json.JSONDecodeError:
return {"case": case["id"], "ok": False, "why": "invalid_json"}
want = case["expect"]
if norm(got.get("label")) != norm(want["label"]):
return {"case": case["id"], "ok": False, "why": "label",
"got": got.get("label"), "want": want["label"]}
fields, gotf = want.get("fields", {}), (got.get("fields") or {})
for k, v in fields.items():
if norm(gotf.get(k)) != norm(v):
return {"case": case["id"], "ok": False, "why": f"field:{k}",
"got": gotf.get(k), "want": v}
return {"case": case["id"], "ok": True, "why": "pass"}Two failure reasons are distinguishable here and that distinction is the whole point:
invalid_jsonis a format drift,labelorfield:*is a content drift.Step 4: Append every run to a ledger and exit non-zero
The ledger is one JSON object per line, append-only, with arun_idderived from the system prompt, model id, and base URL. Change any of those and the id changes, which keeps old and new comparisons honest.def run():
run_id = hashlib.sha256(
(SYSTEM + MODEL + BASE).encode()).hexdigest()[:12]
rows =
[] for line in GOLDEN.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
case = json.loads(line)
raw, err = "", None
for attempt in range(3):
try:
raw = call(case["input"])
break
except Exception as exc: # timeout, 429, DNS, 5xx
err, raw = repr(exc), ""
time.sleep(2 ** attempt)
if not raw:
rows.append({"case": case["id"], "ok": False,
"why": "transport", "err": err})
continue
r = score(case, raw)
r["raw"] = raw[:400]
rows.append(r)
total, passed = len(rows), sum(r["ok"] for r in rows)
rate = round(passed / total, 3) if total else 0.0
record = {"run_id": run_id, "ts": int(time.time()), "total": total,
"passed": passed, "rate": rate, "rows": rows}
with LEDGER.open("a") as fh:
fh.write(json.dumps(record) + "\n")
print(f"{passed}/{total} ({rate:.0%}) run_id={run_id}")
floor = float(os.environ.get("PASS_FLOOR", "0.9"))
return 0 if total and rate >= floor else 1if __name__ == "__main__":
sys.exit(run())Exit codes matter more than dashboards when you are alone. Non-zero from this script can page you, block a deploy, or simply stop a scheduled job from marking the day green.
Step 5: Run it on a schedule without paying for a scheduler
On a free-tier host, cron is the cheapest scheduler you have. Secrets stay in a.envfile that the job sources, so the crontab itself contains no keys.# /srv/app/run-golden.sh
set -euo pipefail
cd /srv/app
set -a; . ./.env; set +a
python3 golden_run.py >> /var/log/golden.log 2>&1# off-peak, once a day; adjust to the hour your host is quietest
17 4 * * * /srv/app/run-golden.shCompare the last two runs with one command. The
jqfilter prints only the cases that flipped.tail -n 2 ledger.jsonl \
| jq -c '{run_id, ts, rate, fails: [.rows[] | select(.ok == false) | .case]}'Free-tier containers often sleep between requests. A 30-second timeout and a three-attempt backoff absorb most cold-start noise, and the
transportreason keeps that noise out of your content-drift count.Reading the ledger: a decision table
Signal in the ledgerLikely causeFirst actionManytransport, few content failuresthrottling, cold start, or a slow windowMove the job off-peak, lower case count, raise timeoutSuddeninvalid_jsonon every caseresponse formatting default changedRe-check the adapter and system prompt before blaming the modelOne field fails on most casesreal content changePin a previous model id if one is offered, or add a normalization stepOne case fails, everything else passesa genuinely hard exampleRe-read the expected value; the baseline may be wrong
A single bad run is not a verdict. Two consecutive runs with the same failure set is the threshold worth acting on.Limitations and who should skip this
Exact-match scoring is brittle by design and wrong for open-ended generation. Do not point this harness at a summarizer, a chat assistant, or anything whose value is in the phrasing.A 20-to-40 case set cannot detect rare failures, and it gives no statistical confidence. Teams that need eval coverage, graders, or per-release reports should use a real evaluation platform instead.
The adapter is single-turn. Agents that call tools across multiple round trips need a different harness, or at least a recorded-transcript replay layer in front of this one.
Finally, the runtime budget is real. Free-tier quotas, sleep windows, and timeouts change; the ledger tells you when they did, but it will not make them go away.
Where MonkeyCode fits in this loop
Disclosure: This article was prepared as part of MonkeyCode's product outreach.The harness is endpoint-agnostic, which is the point. In my own loop, MonkeyCode is the endpoint behind
MODEL_BASE_URLand the host that runs the cron job, because its free model access and free server option keep a nightly regression run off the invoice while the project is still pre-revenue. The operator describes the free tier as including a substantial token allowance — on the order of ten million tokens — plus a free server option; verify the current terms on the project page before you size a golden set against it, since quotas and limits move faster than articles do. The project is open source, so the transport adapter is the only part you may need to touch.If the loop is useful, the smallest honest start is one line in
golden.jsonl: take a real failure your users already reported, freeze the correct output, and let the ledger watch it for you.#ai #testing #opensource #devops #software #coding #development #engineering #inclusive #community
Detect Silent Model Drift With a Stdlib-Only Golden Set Harness -
A generated patch rarely fails review because of formatting, naming, or even local correctness. It fails because the reviewer cannot reconstruct the failure it was meant to fix, the files it read, and the decision it quietly made. A senior engineer paired on such a patch will often ignore the diff for the first ten minutes and walk a fixed ladder of questions instead. Once those answers are written down, the patch becomes either clearly acceptable or clearly wrong, and the debate collapses into a record.
The ladder matters more than the lint
Lint, type checks, and unit tests answer questions about the code exactly as written. They say nothing about the reasoning that produced the change, which is where generated patches go wrong most often. A patch can pass every check and still edit a module nobody asked it to touch, or hard-code an assumption that only held inside the prompt. The review therefore needs two distinct gates: a mechanical gate for the artifact, and a conversational gate for the intent behind it.The five questions, in order
- Reproduction first. Can the failure be reproduced from a clean checkout without the patch? Acceptable evidence is a command, an exit code, and one log line. If the honest answer is "probably", the session ends before the diff is opened.
- Read set second. Which files, tables, or endpoints does the patch read outside the module it edits? Acceptable evidence is a grep, a trace, or an explicit import list. This catches the most common dead end: a patch that solves the stated problem by reaching into three unrelated packages.
- Contract third. Which existing behavior does the patch promise to preserve, and where is that promise tested? Acceptable evidence is the name of an existing test that still passes. Silence here usually means the patch is a rewrite wearing a bugfix costume.
- Rollback fourth. If the patch turns out wrong in production, what single command or flag reverses it? Acceptable evidence is the revert path itself, not a plan to write one later.
- Owner fifth. Who answers for this decision in six months, and where is that recorded? Acceptable evidence is a name plus a file, usually the same decision record the session produces.
Questions one and two eliminate most candidates within minutes. Questions three through five decide whether the survivor is mergeable today or needs a follow-up ticket instead.
Running the session as a reproducible artifact
- Scaffold the session directory and freeze the candidate. Copy the patch and hash it, so the reviewed bytes cannot drift during the conversation.
- Record the base revision. Without it, no later reader can tell which tree the patch was generated against.
- Answer the ladder in a machine-readable file. Free text in a chat thread is not a decision record; JSON or YAML is.
- Gate on completeness before anyone runs the tests. An unanswered question should stop the pipeline, not scroll past it.
# questions.yml - the ladder as data
- id: q1_repro
text: "Can the failure be reproduced from a clean checkout without the patch?"
evidence: "command + exit code + one log line"
- id: q2_readset
text: "Which files or tables does the patch read outside the edited module?"
evidence: "grep output or trace excerpt"
- id: q3_contract
text: "Which existing behavior must be preserved, and which test proves it?"
evidence: "existing test name + passing run"
- id: q4_rollback
text: "What is the single command or flag that reverses this change?"
evidence: "revert command, not a plan"
- id: q5_owner
text: "Who owns this decision, and where is it recorded?"
evidence: "name + decision file path"#!/usr/bin/env bash
# review-session.sh <session-dir> <candidate.patch>
set -euo pipefailSESSION_DIR="${1:?usage: review-session.sh <session-dir> <patch-file>}"
PATCH="${2:?usage: review-session.sh <session-dir> <patch-file>}"
mkdir -p "$SESSION_DIR"cp "$PATCH" "$SESSION_DIR/candidate.patch"
sha256sum "$SESSION_DIR/candidate.patch" | awk '{print $1}' > "$SESSION_DIR/candidate.sha256"
git rev-parse HEAD > "$SESSION_DIR/base-rev.txt" 2>/dev/null || echo "no-repo" > "$SESSION_DIR/base-rev.txt"
printf '{}\n' > "$SESSION_DIR/answers.json"cp questions.yml "$SESSION_DIR/questions.yml"
echo "session scaffolded at $SESSION_DIR"#!/usr/bin/env bash
# gate.sh <session-dir> - blocks until every ladder entry has an answer
set -euo pipefailSESSION_DIR="${1:?usage: gate.sh <session-dir>}"
jq -e 'all(.[]; (.answer | type) == "string" and (.answer | length) > 0)' \
"$SESSION_DIR/answers.json" > /dev/null \
|| { echo "BLOCKED: unanswered question in the ladder"; exit 1; }jq -r 'to_entries[] | "\(.key)\t\(.value.answer)\t\(.value.evidence)"' \
"$SESSION_DIR/answers.json" > "$SESSION_DIR/decision.tsv"echo "PASS: decision record written to $SESSION_DIR/decision.tsv"
The two scripts are deliberately boring, because the value is in the ordering. The hash pins the artifact, the base revision pins the context, and the gate refuses to let a partial conversation look like a completed review.
Where a free model and a free server fit
A candidate patch has to come from somewhere before the ladder can be applied to it. MonkeyCode offers free model access for producing that candidate and a free server option for running the reproduction in a disposable environment rather than on a reviewer's laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. In this workflow the product participates in two narrow steps only: drafting the patch that the session will interrogate, and hosting the clean checkout where questions one and four get verified. Availability, allowances, and terms for both options can change, so confirm the current details on MonkeyCode's own documentation before depending on them for anything scheduled.Keeping the reproduction off the reviewer's machine matters for a mundane reason. A clean checkout on a disposable host proves the failure exists without local caches, editor plugins, or half-installed dependencies. The session log then records a host and a revision instead of a workstation nobody else can reach.
The decision table the session produces
Ladder outcomeActionWho decidesq1 or q2 has no evidenceReject and return to the requesterReviewerq1, q2 answered; q3 missingAccept the fix, open a separate preservation ticketReviewer + ownerq1 to q4 answered; q5 missingAssign an owner in the decision file, then mergeTech leadAll five answered with evidenceMerge, keep the session directory as the recordPairLimitations and who should not use this
This workflow assumes a repository with a reproducible entry point and at least one automated test that can run headlessly. It is poor value for typo fixes, dependency bumps, and one-line configuration changes, where the ladder costs more than the patch. Teams shipping one candidate patch per week will find the ten-to-fifteen minute session overhead hard to justify, and teams handling regulated data should confirm vendor terms and internal policy before pushing any source to a hosted free service.The free server option is a disposable environment, and it should never hold production credentials, customer data, or long-lived secrets. The ladder also does not replace a security review, a license scan, or a human who understands the domain the patch touches. Treat the recorded decision file as evidence of process, not as proof of correctness.
Dead ends worth naming
The most expensive dead end is a patch that reproduces the failure only after an uncommitted local edit, because question one then has no clean answer. The second is a read set discovered after the merge, which usually appears as a surprise import in a diff nobody re-reads. Both are cheap to catch at the start of a session and expensive to catch three weeks later during an incident review.If a team wants to try the pattern without provisioning anything, the free model access and free server option through MonkeyCode are a reasonable place to run one session end to end and see whether the ladder earns its keep.#ai #programming #codequality #devtools #software #coding #development #engineering #inclusive #community
A Senior Asks Five Questions Before Reading Your Generated Patch
