A fluent patch is not a pass. A free server that happens to boot is not a pass either. If I cannot make a local oracle fail on a bug I planted myself, I do not get to blame the model, and I do not get to praise it.
That is the conclusion. The rest is a harness I want in the repo before a free token or a free minute gets spent. This draft does not report a live run, because I did not execute one for it.
You know the scene. The assistant nods, the diff looks tidy, and the slice is still wrong. Who gets the lecture, the model or the missing test?
Blame is a warm drink. It does not fail a build. I want the check to fail first, on purpose, with a command I can rerun when I am bored and suspicious. No leaderboard, and no borrowed authority. Just an exit code that was already red before any model was allowed to speak.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project, and two availability claims here are operator-supplied rather than measurements from 13 October 2026: free model access, and a free server option.
I was also told the free model side includes a grant on the order of 10 million tokens. Confirm that figure, which models it actually covers, and the server terms on the current project page before you plan a week around them. I will not invent model names, hardware, durations, or a win rate.
The call site still has to live in git, next to the plants, or the evaluation is a chat you cannot replay. Free access is a budget for that replay. It is not evidence the patch was right. Delete the product name and the stop-rule still applies to any client you already have.
People keep saying the models got sharper while the bugs stayed put. Maybe they did. Sharpness without a red oracle is a brighter flashlight in a room you never filled with smoke. I am not grading a summer of announcements. I am asking whether my own gate can fail a bug I already understand.
Plant something boring
Clever bugs flatter the person who planted them. Boring bugs audit the gate. I keep one function that returns one element too many, the kind a tired reviewer can spot, because that is the point.
If the oracle cannot see a bug a tired human can see, why would I trust it on the bugs I miss at 1 a.m.? The plant stays small so the failure is about the gate, not about my talent for hiding a defect in a maze.
# harness/plants.py
# Proposal. Not a recorded run.
from dataclasses import dataclass
@dataclass(frozen=True)
class Plant:
name: str
buggy: str
fixed: str
TAKE = Plant(
name='take-off-by-one',
buggy='''def take(xs, n):
return xs[: n + 1]
''',
fixed='''def take(xs, n):
return xs[:n]
''',
)
The test is equally dull. Dull is a feature, not a lack of ambition. It names one boundary and refuses to negotiate.
# tests/test_take.py
from take import take
def test_take_does_not_overshoot():
assert take(['a', 'b', 'c'], 2) == ['a', 'b']
A writer script keeps me from pasting the bug by hand and then forgetting which variant is on disk. Hands are how plants get quietly fixed before the model ever sees them.
# harness/write_plant.py
# Proposal. Overwrites take.py from the plant table.
import argparse
from plants import TAKE
BODIES = {'buggy': TAKE.buggy, 'fixed': TAKE.fixed}
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument('--name', required=True)
parser.add_argument('--variant', choices=sorted(BODIES), required=True)
args = parser.parse_args()
if args.name != TAKE.name:
raise SystemExit('unknown plant')
with open('take.py', 'w', encoding='utf-8') as fh:
fh.write(BODIES[args.variant])
if __name__ == '__main__':
main()
Write the buggy body into place and let pytest speak before you do. If you feel an urge to peek at a completion first, that urge is the experiment trying to cheat.
python harness/write_plant.py --name take-off-by-one --variant buggy
python -m pytest -q tests/test_take.py
echo oracle_exit=$?
If that exit is 0, stop. The plant is invisible. Asking a free model to review an invisible bug is a play, not a measurement. Fix the test, and do not touch the prompt.
A smoke alarm you have never held to smoke can still beep in a drill. The red pytest is the smoke. Without it I am admiring the plastic cover and calling it safety.
Only then may the model look
The prompt stays poorer than the test file. I do not paste the expected list. I do not ask whether the function is good, because that adjective is how a yes gets hired as an engineer.
The instruction I send is plain. One function is attached. A test named test_take_does_not_overshoot fails. Return either a line that starts with REJECT and one sentence, or a unified diff that changes only take. Do not claim the tests were run. Save the reply next to the plant name. Feeling is not a column.
# harness/classify.py
# Proposal. A malformed reply is a miss, not a cue to retry.
def classify(reply: str) -> str:
text = reply.strip()
if text.startswith('REJECT:'):
return 'reject'
if text.startswith('---') or text.startswith('diff '):
return 'diff'
return 'malformed'
Then the mechanical half, still on the laptop, still too boring to argue with:
python harness/extract_diff.py reply.txt > candidate.diff
patch -p0 < candidate.diff
python -m pytest -q tests/test_take.py
echo after_model_exit=$?
Two exits. They do not get to shake hands. The first had to be non-zero or the day is void. The second may be zero, and that is a different sentence.
A REJECT on a function the oracle already hates can be a real hit. The model refused to launder a bad slice. A minimal diff that goes green is a different hit. Would you average those and call the average quality? I will not. Unlike events do not become one fact just because they share a spreadsheet.
If patch refuses the diff, I do not retry inside the scorer. A diff that does not apply is a miss. Retry-until-apply is another study, and it will eat a grant while wearing a lab coat.
The ledger is the third gate
Free tokens disappear in the rephrase. You paste the traceback, you ask again, and you call it iteration. The meter calls it lunch.
This ledger is local, and it should stay proud of that. If the provider meter disagrees, the provider wins. Write the gap down. Do not litigate it in the README and pretend the argument is a measurement.
# harness/ledger.py
# Proposal. Counts what this process tried to spend.
class Ledger:
def __init__(self, ceiling: int):
self.ceiling = ceiling
self.spent = 0
def charge(self, prompt_tokens: int, completion_tokens: int) -> None:
self.spent += prompt_tokens + completion_tokens
if self.spent > self.ceiling:
raise SystemExit(
f'stop: spent {self.spent} > ceiling {self.ceiling}'
)
Set the ceiling to a slice, not to the folklore of the whole grant. A few thousand tokens will tell you whether the loop is bounded. Ten million, if the project page still says that when you look, is headroom. It is not a result I measured, and it is not a promise I can extend.
Quotas move. A method that only works if a published number never changes is not a method. Re-read the page the day you run the card, and write the number you actually saw into the notebook next to the date.
python harness/run_eval.py --plants harness/plants --ceiling 4000 --dry-run
The dry-run flag prints the prompt and charges nothing. I want it the way I want a seatbelt in a parking lot. An evaluation you cannot rehearse will get rehearsed on the grant, by accident, late, when you are sure this one extra call does not count.
The free server witnesses. It does not judge.
I may use the free server only after the local oracle is already red, and only to ask a narrower question. Does the same pytest exit match on that box? That is a witness statement about the room. It does not grade taste, tone, or how confident the completion sounded.
Laptop red and server green means portability, not genius. Both red means the patch missed, or the plant is still in the tree. Both green before the model spoke means the day is invalid, so go back and make smoke.
I am not asking a remote box whether a model is smart. Smart is not a command. Pytest is a command. If the documented runner is different from the placeholder below, keep the exit code and throw my hostname away.
# Placeholder only. Use the documented free-server run command.
# example.test is not a product host.
ssh [email protected] -- python -m pytest -q tests/test_take.py
echo witness_exit=$?
Keep the exit code. Ignore any story a boot splash tries to sell. A machine that started is a machine that started. That sentence feels obvious until a green banner lands in a screenshot and everybody in the thread relaxes.
Read the card like gauges
When the card comes back I read it the way a mechanic reads gauges, not the way a fan reads a scoreline. Oracle not red? The card is void. No argument, no partial credit, no interesting qualitative notes.
Oracle red and the reply malformed? A miss, and the ledger still charged, which is the lesson hiding in the annoyance. Oracle red, diff applies, pytest green, witness matches? That is the only quiet pass I trust, and only for this plant.
Oracle red, and the model calls the buggy function fine? False praise. That cell matters more to me than a pretty repair. A tool that blesses a known-bad slice will bless the next one when I am tired and late. Where does a free model tend to look strong on a pack like this? On the boring repair, once the oracle has already named the failure. Where does it break? On false praise, on diffs that do not apply, and on any day the local check was blind. The free server breaks the story when its exit disagrees with the laptop, and it proves nothing when you boot it first and call the boot a result.
Four counts sit side by side, and they never get blended. Oracle-fail rate on plants, which must be total or you throw the day out. Repair rate on plants the oracle actually caught. False-praise rate. Witness mismatches between the laptop and the free server.
I am leaving those cells empty on purpose. No timings. No win rate. No claim that one stack beat another. You fill them on your plants, and you keep the raw exits next to the commit. An empty card is stricter than a handy number I could have typed.
Who should walk away
Skip this if you need a procurement bake-off. A toy slice will flatter whoever has seen an off-by-one in training, and it will teach you almost nothing about a 400-file service. Skip it for auth, crypto, or any hazard a unit test does not name.
A green take() will not make a token check safe. Skip it if the free tier forbids scripted calls. A ledger is not a pardon, and I am not suggesting you route around a limit. Skip it if you wanted a leaderboard by Friday. You wanted a mood, and moods do not belong in the README.
Two limits sit in the design, and they will not politely stay small. Leakage is the first. Paste the expected list into the prompt and a model can patch the example instead of the function. Keep the prompt hungrier than the test file.
Scale is the second. One plant is a fire drill, not a map of the repo. Restage bugs you have already fixed, oracle included, names stripped so you are not grading memory of your own commit message. A bug you cannot restage is an anecdote with better lighting. Treat it as a story. Do not enter it on the card.
If you point this same stop-rule at MonkeyCode's free model, and only afterward at the free server, read the current docs first. Confirm the token grant and the server limits yourself. Run the dry flag before you spend anything. I would rather you distrust the grant than trust this paragraph.#testing #python #ai #programming #software #coding #development #engineering #inclusive #community
I Made the Oracle Fail First
