An agent that feels instant on a laptop can feel broken the moment its model call crosses a network boundary, and the postmortem usually blames the model. The regression more often comes from three numbers nobody recorded before the move: cold-start latency, payload size leaving the device, and behaviour when the remote seat is unreachable. The scenario below is illustrative rather than a customer story, but the failure shape is common enough to plan around.
This article builds a small probe that records those three numbers for any endpoint pair and turns them into a residency decision. It assumes you already have an agent that runs end to end and you are choosing where the model call happens. Nothing here depends on one vendor, and every threshold below is a starting point you should retune for your own product.
Why three measurements beat one benchmark
Public leaderboards measure model quality on fixed prompts, and they say almost nothing about the machine your agent actually runs on. Residency is a property of your process, your network path and your prompt size, so it has to be measured in place. A single latency number is also easy to game: a warm local runtime can win on the median and lose badly on the first request after a laptop wakes up.
The probe therefore records a distribution rather than a point estimate, and it pairs that distribution with two non-latency signals. The payload audit answers what content leaves the device, which matters when prompts carry repository context or customer records. The offline check answers whether the agent degrades into a queue or into an exception, and that difference decides how much of the fallback path you have to build.
Step 1: Fix one request shape for both endpoints
The comparison is only meaningful when both sides receive the same bytes, so define one request body and reuse it. Keep the system prompt and user prompt in files, and let environment variables supply the base URLs. This also makes the probe cheap to rerun whenever a runtime updates.
mkdir -p prompts
printf 'Summarise the three functions in this diff.\n' > prompts/short.txt
export LOCAL_BASE_URL="http://127.0.0.1:8080/v1/complete"
export REMOTE_BASE_URL="https://your-endpoint.example/v1/complete"
The two URLs are placeholders, and the script below does no vendor-specific shaping. If your local runtime expects a different JSON envelope, adapt build_body once and keep the rest identical. That single edit point is what makes the numbers comparable later.
Step 2: Measure first body byte, with explicit honesty about it
Time to first token is the metric developers feel, but most HTTP clients only expose time to first body byte. The probe reads exactly one byte and stops, then labels the result as an approximation. It also reports p50 and p95 separately, because tail behaviour is where remote seats and cold local runtimes diverge.
The script also counts failures instead of hiding them behind a mean. A run that times out is data, and a remote seat that fails two of seven attempts changes the architecture conversation more than a hundred milliseconds of median latency.
#!/usr/bin/env python3
"""residency_probe.py - reference implementation, not a benchmark suite.
Records first-body-byte latency, payload size, secret-pattern hits and
forced-offline behaviour for up to two endpoints. Every number it prints
is yours to collect; no vendor performance is asserted here.
"""
from __future__ import annotations
import argparse
import json
import os
import re
import socket
import statistics
import time
import urllib.error
import urllib.request
SECRET_PATTERNS = {
"provider_key": re.compile(r"sk-[A-Za-z0-9]{16,}"),
"bearer_value": re.compile(r"(?i)bearer[ ]+[A-Za-z0-9._-]{12,}"),
"aws_secret": re.compile(r"(?i)aws_secret_access_key[ ]*[:=]"),
}
def build_body(prompt: str, system: str) -> bytes:
return json.dumps({"system": system, "prompt": prompt}).encode("utf-8")
def scan_for_secrets(raw: bytes) -> list[str]:
text = raw.decode("utf-8", errors="replace")
hits = [name for name, pat in SECRET_PATTERNS.items() if pat.search(text)]
for key in ("API_KEY", "ACCESS_TOKEN", "SECRET"):
value = os.environ.get(key, "")
if len(value) >= 12 and value in text:
hits.append("env:" + key)
return hits
def first_body_byte_ms(url: str, body: bytes, timeout: float) -> float:
req = urllib.request.Request(
url, data=body, headers={"Content-Type": "application/json"}
)
start = time.perf_counter()
with urllib.request.urlopen(req, timeout=timeout) as resp:
resp.read(1) # first body byte, NOT first token
return (time.perf_counter() - start) * 1000.0
def probe(url: str, body: bytes, attempts: int, timeout: float) -> dict:
samples, failures = [], 0
for _ in range(attempts):
try:
samples.append(first_body_byte_ms(url, body, timeout))
except (urllib.error.URLError, socket.timeout, TimeoutError, ConnectionError):
failures += 1
ordered = sorted(samples)
def pct(p: float) -> float | None:
if not ordered:
return None
idx = min(len(ordered) - 1, int(round(p * (len(ordered) - 1))))
return round(ordered[idx], 1)
return {
"url": url,
"attempts": attempts,
"failures": failures,
"p50_ms": pct(0.50),
"p95_ms": pct(0.95),
"mean_ms": round(statistics.fmean(samples), 1) if samples else None,
}
def offline_behaviour(fallback) -> dict:
"""Point at the discard port, then check the fallback still returns work."""
dead = "http://127.0.0.1:9/v1/complete"
started = time.perf_counter()
try:
first_body_byte_ms(dead, b"{}", timeout=2.0)
reached = True
except (urllib.error.URLError, socket.timeout, TimeoutError, ConnectionError):
reached = False
result = fallback()
return {
"endpoint_reachable": reached,
"fallback_returned": result is not None,
"fallback_seconds": round(time.perf_counter() - started, 3),
}
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--prompt", default="prompts/short.txt")
ap.add_argument("--attempts", type=int, default=7)
ap.add_argument("--timeout", type=float, default=20.0)
ap.add_argument("--out", default="residency_report.json")
args = ap.parse_args()
prompt = open(args.prompt, encoding="utf-8").read().strip()
body = build_body(prompt, system="residency-probe")
report = {
"payload_bytes": len(body),
"payload_secret_hits": scan_for_secrets(body),
"endpoints": [],
"offline": offline_behaviour(lambda: {"queued": True}),
}
for name in ("LOCAL_BASE_URL", "REMOTE_BASE_URL"):
url = os.environ.get(name)
if not url:
continue
row = probe(url, body, args.attempts, args.timeout)
row["name"] = name
report["endpoints"].append(row)
with open(args.out, "w", encoding="utf-8") as fh:
json.dump(report, fh, indent=2)
print(json.dumps(report, indent=2))
if __name__ == "__main__":
main()
Run it twice: once with the local runtime freshly started, and once after it has served a few requests. Those two runs are the cheapest way to see cold-start cost without building a load generator.
Step 3: Audit what actually leaves the device
The payload audit is deliberately blunt, and it is not a data-loss-prevention product. It flags obvious credential shapes and any long environment value that leaked into the serialised body, which catches the common accident of pasting a token into a system prompt. A hit is a stop sign for a remote seat and a warning for a local one, because local runtimes log too.
The byte count matters as much as the hits. A prompt carrying an entire diff can be orders of magnitude larger than a chat message, and upload time on a home connection scales with it. Record the byte count next to the latency figures, or you will attribute a bandwidth problem to model speed.
Step 4: Force the offline path on purpose
The probe points at a dead port and then calls the same fallback your agent would call. If the fallback raises instead of returning queued work, the agent needs a fallback contract before any residency change ships. If it returns queued work in a few seconds, the offline story is already credible.
This check is cheap and it prevents the most expensive migration surprise. A remote seat that disappears during a demo is survivable when work queues locally and can be resumed. It is fatal when the whole run lives in a request handler that has already returned.
A decision table you can argue with
The thresholds below are operator-tunable starting points rather than published guidance. Fill the middle columns with your own probe output and use the last two as the interpretive lens.
SignalWhat local-first tends to giveWhat a free server tends to giveRead it asWarm p95 first body byteLow and stable once the runtime is pinnedDepends on queueing at the shared endpointLocal wins on steady interactive loopsCold startLong first request after sleep or restartEndpoint is already warmRemote wins on sporadic usePayload bytes off deviceZero for the model callFull prompt, including contextSensitive context argues for localSecret-pattern hitsStill a bug, still loggedBlocks the send until redactedAny hit stops the remote optionOffline behaviourUsable if you persist stateFails closed without a queueQueue design is the real workConcurrency shapeBounded by one machineBursts absorbed elsewhereSpiky workloads favour remote
The table is a discussion device, not a verdict. Two teams with identical numbers can land on different rows because their data classification and support burden differ.
Where a free server genuinely wins
A recurring discussion in developer circles concerns how many intermediaries sit between a developer and a model, and residency is the concrete version of that question. The honest answer is that self-hosting is sometimes the wrong trade, especially when the local machine cold-starts constantly or has no accelerator at all. Spiky workloads, shared team artifacts and short-lived experiments all favour a remote seat that is already warm.
MonkeyCode, an open-source coding agent project, offers free model access and a free server option, which makes it a reasonable second column for the probe above. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Token allowances and server terms change, so read the project page for the current limits before you design around them, and never assume a free tier is permanent.
The useful pattern is to run the probe against a free remote seat and against your laptop on the same day with the same prompt file. That produces a defensible comparison instead of an intuition, and it costs one environment variable.
Limitations and who should not use this
The probe measures first body byte, not time to first token, and the two diverge when a server streams headers early. It is a single-machine instrument, so it is not a load test and it will not reveal behaviour under ten concurrent runs. The secret scan is a regex tripwire, not a compliance control, and it will miss bespoke credential formats.
Teams with hard data-residency obligations should keep the model call local and treat the remote column as unavailable rather than slow. Interactive products with tight latency budgets should not adopt a shared endpoint without a warm-up strategy and a measured tail. Anyone expecting deterministic throughput from a free tier should build the queue first and treat the server as an accelerator, not a guarantee.
If you want a second column without provisioning anything, point the probe at a free remote seat first and keep the local runtime as the fallback. The numbers you collect will tell you more than any migration plan written before them.#ai #localfirst #python #architecture #software #coding #development #engineering #inclusive #community
Measuring Before Migrating: A Residency Smoke Test for Local-First Agents and Free Servers
