📌 El futuro llegó: Elon Musk presenta el auto volador Roadster de Tesla y SpaceX que desafía a la gravedad
Por redpres.com
Tags | 🏷️ #Tesla #Roadster #SpaceX #ElonMusk #Propulsores #Superdeportivo #Texas
Search
4 results for “respR”
-
-
📌 Cómo leer las tasas de interés de los bancos centrales sin ser economista
Tags | 🏷️ #TasasDeInterés #BancosCentrales #Forex #ReservaFederal #BCE #Trade #Trading
-
Entre nós, guineenses, o “agora” tende a absorver tudo: a nomeação, a reaproximação, a oportunidade de entrar no Governo. O passado serve para acusar, raramente para aprender. Opinião de Umaro Djau
https://www.publico.pt/2026/10/16/mundo/opiniao/guinebissau-transicao-vira-forma-governar-2191663 #notícias -
Detect Silent Model Drift With a Stdlib-Only Golden Set Harness
A hosted model can change between two of your deploys, and nothing in your lockfile moves. Your unit tests still pass, your container still builds, and your users still get slightly different JSON than they got last Tuesday. The practical fix is not a bigger test suite; it is a frozen golden set, a deterministic scorer, and an append-only ledger that fails the run when the pass rate drops below a floor.This article builds exactly that, using only the Python standard library. It runs on a free-tier host, it costs one file of input data, and it answers a single narrow question: did the contract my product depends on stop holding?
Why ordinary tests miss model drift
Application tests assert on code you control. A model response is an input from outside your repository, and it arrives as free text that happens to be parseable.When a provider adjusts decoding defaults, swaps capacity, or routes you to a different checkpoint, the shape of the output can shift while the API status code stays
200. Nothing throws. The failure surfaces later, as a support message about a field that used to be filled in.A golden set inverts that. Instead of asserting on the current behavior, it asserts on a small number of hand-checked examples that were correct on a specific day, and it keeps the receipts.
Step 1: Freeze a small golden set as JSONL
The input file is plain JSONL, one contract per line. Keep it small on purpose: 20 to 40 cases is enough to catch a formatting or label flip, and it fits inside a free-tier request budget.{"id":"invoice_date","input":"Invoice paid 2026-03-04, net 30 days.","expect":{"label":"invoice","fields":{"due_date":"2026-04-03"}}}
{"id":"refund_request","input":"I want my money back for order 8812, it never arrived.","expect":{"label":"refund","fields":{"order_id":"8812"}}}
{"id":"no_action","input":"Thanks, the docs answered my question.","expect":{"label":"none","fields":{}}}Every line has three parts: an
idyou will grep for in the ledger, the rawinputyou send, and theexpectobject you compare against. Do not paste generated examples into this file. If you did not read the output and agree with it, it does not belong in the baseline.Step 2: Write a transport adapter that assumes nothing
The runner below assumes an OpenAI-style/chat/completionsrequest body. That assumption is isolated in one function so you can replace it when your endpoint differs.#!/usr/bin/env python3
"""golden_run.py - regression run against a chat endpoint. Stdlib only."""
import json, os, sys, time, hashlib, pathlib, urllib.requestBASE = os.environ["MODEL_BASE_URL"].rstrip("/")
KEY = os.environ["MODEL_API_KEY"]
MODEL = os.environ["MODEL_ID"]
GOLDEN = pathlib.Path(os.environ.get("GOLDEN", "golden.jsonl"))
LEDGER = pathlib.Path(os.environ.get("LEDGER", "ledger.jsonl"))SYSTEM = (
"You are a strict extractor. Return ONLY minified JSON shaped as "
'{"label": string, "fields": object}. Omit absent fields. '
"No prose, no markdown fences."
)def call(text, timeout=30):
body = json.dumps({
"model": MODEL,
"temperature": 0,
"messages": [{"role": "system", "content": SYSTEM},
{"role": "user", "content": text}],
}).encode()
req = urllib.request.Request(
f"{BASE}/chat/completions", data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {KEY}"},
)
with urllib.request.urlopen(req, timeout=timeout) as r:
payload = json.loads(r.read())
return payload["choices"][0]["message"]["content"]Keep
temperatureat zero when the endpoint supports it. If the parameter is rejected or ignored, note that in your README rather than deleting it, because a later comparison depends on knowing what you actually sent.Step 3: Score the contract, not the prose
Scoring is exact after light normalization: trim, lowercase, collapse whitespace. That is deliberately unforgiving, because the goal is to detect change, not to grade writing.def norm(v):
return " ".join(v.strip().lower().split()) if isinstance(v, str) else vdef score(case, raw):
try:
got = json.loads(raw)
except json.JSONDecodeError:
return {"case": case["id"], "ok": False, "why": "invalid_json"}
want = case["expect"]
if norm(got.get("label")) != norm(want["label"]):
return {"case": case["id"], "ok": False, "why": "label",
"got": got.get("label"), "want": want["label"]}
fields, gotf = want.get("fields", {}), (got.get("fields") or {})
for k, v in fields.items():
if norm(gotf.get(k)) != norm(v):
return {"case": case["id"], "ok": False, "why": f"field:{k}",
"got": gotf.get(k), "want": v}
return {"case": case["id"], "ok": True, "why": "pass"}Two failure reasons are distinguishable here and that distinction is the whole point:
invalid_jsonis a format drift,labelorfield:*is a content drift.Step 4: Append every run to a ledger and exit non-zero
The ledger is one JSON object per line, append-only, with arun_idderived from the system prompt, model id, and base URL. Change any of those and the id changes, which keeps old and new comparisons honest.def run():
run_id = hashlib.sha256(
(SYSTEM + MODEL + BASE).encode()).hexdigest()[:12]
rows =
[] for line in GOLDEN.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
case = json.loads(line)
raw, err = "", None
for attempt in range(3):
try:
raw = call(case["input"])
break
except Exception as exc: # timeout, 429, DNS, 5xx
err, raw = repr(exc), ""
time.sleep(2 ** attempt)
if not raw:
rows.append({"case": case["id"], "ok": False,
"why": "transport", "err": err})
continue
r = score(case, raw)
r["raw"] = raw[:400]
rows.append(r)
total, passed = len(rows), sum(r["ok"] for r in rows)
rate = round(passed / total, 3) if total else 0.0
record = {"run_id": run_id, "ts": int(time.time()), "total": total,
"passed": passed, "rate": rate, "rows": rows}
with LEDGER.open("a") as fh:
fh.write(json.dumps(record) + "\n")
print(f"{passed}/{total} ({rate:.0%}) run_id={run_id}")
floor = float(os.environ.get("PASS_FLOOR", "0.9"))
return 0 if total and rate >= floor else 1if __name__ == "__main__":
sys.exit(run())Exit codes matter more than dashboards when you are alone. Non-zero from this script can page you, block a deploy, or simply stop a scheduled job from marking the day green.
Step 5: Run it on a schedule without paying for a scheduler
On a free-tier host, cron is the cheapest scheduler you have. Secrets stay in a.envfile that the job sources, so the crontab itself contains no keys.# /srv/app/run-golden.sh
set -euo pipefail
cd /srv/app
set -a; . ./.env; set +a
python3 golden_run.py >> /var/log/golden.log 2>&1# off-peak, once a day; adjust to the hour your host is quietest
17 4 * * * /srv/app/run-golden.shCompare the last two runs with one command. The
jqfilter prints only the cases that flipped.tail -n 2 ledger.jsonl \
| jq -c '{run_id, ts, rate, fails: [.rows[] | select(.ok == false) | .case]}'Free-tier containers often sleep between requests. A 30-second timeout and a three-attempt backoff absorb most cold-start noise, and the
transportreason keeps that noise out of your content-drift count.Reading the ledger: a decision table
Signal in the ledgerLikely causeFirst actionManytransport, few content failuresthrottling, cold start, or a slow windowMove the job off-peak, lower case count, raise timeoutSuddeninvalid_jsonon every caseresponse formatting default changedRe-check the adapter and system prompt before blaming the modelOne field fails on most casesreal content changePin a previous model id if one is offered, or add a normalization stepOne case fails, everything else passesa genuinely hard exampleRe-read the expected value; the baseline may be wrong
A single bad run is not a verdict. Two consecutive runs with the same failure set is the threshold worth acting on.Limitations and who should skip this
Exact-match scoring is brittle by design and wrong for open-ended generation. Do not point this harness at a summarizer, a chat assistant, or anything whose value is in the phrasing.A 20-to-40 case set cannot detect rare failures, and it gives no statistical confidence. Teams that need eval coverage, graders, or per-release reports should use a real evaluation platform instead.
The adapter is single-turn. Agents that call tools across multiple round trips need a different harness, or at least a recorded-transcript replay layer in front of this one.
Finally, the runtime budget is real. Free-tier quotas, sleep windows, and timeouts change; the ledger tells you when they did, but it will not make them go away.
Where MonkeyCode fits in this loop
Disclosure: This article was prepared as part of MonkeyCode's product outreach.The harness is endpoint-agnostic, which is the point. In my own loop, MonkeyCode is the endpoint behind
MODEL_BASE_URLand the host that runs the cron job, because its free model access and free server option keep a nightly regression run off the invoice while the project is still pre-revenue. The operator describes the free tier as including a substantial token allowance — on the order of ten million tokens — plus a free server option; verify the current terms on the project page before you size a golden set against it, since quotas and limits move faster than articles do. The project is open source, so the transport adapter is the only part you may need to touch.If the loop is useful, the smallest honest start is one line in
golden.jsonl: take a real failure your users already reported, freeze the correct output, and let the ledger watch it for you.#ai #testing #opensource #devops #software #coding #development #engineering #inclusive #community
Detect Silent Model Drift With a Stdlib-Only Golden Set Harness