A maintainer on a small library closed a pull request after the patch looked tidy and the model review sounded confident. The contributor had pasted a long log into chat, received praise, and opened the request without a public failing fixture. On a clean clone of the default branch, the reported command exited zero, so the patch had nothing to prove. That scene is a composite of common review failures, not a claim about one named project.
The same pattern appears when a model reviews a diff before anyone proves the bug on the upstream default branch. Agreement then becomes a style comment about the patch, rather than evidence that the failure existed at all. A useful workflow freezes a public bundle first, then invites a model to comment only on fields inside that bundle. The bundle, not the model's tone, decides whether the patch is ready for a human maintainer.
Freeze a public bundle the model is allowed to see
A review bundle is a directory committed on a scratch branch, or attached to the issue, before any model sees the patch. It holds the failing command, the expected exit status, a fixture file, and a short note about the default-branch commit. It also holds a license check result and a test command copied from the repository's own documentation. Secrets, tokens, and private hostnames stay out of that directory so the packet can be shared with a model.
Fields that make the packet checkable
- The file
base_sha.txt records the upstream commit where the failure was observed, and later reruns must detach at that commit. - The file
exit_status.txt stores the status of the failing command, and a clean default-branch run must reproduce that status. - The file
fixture.sha256 names the public input, so a review cannot swap in a quieter file after the model has spoken. - The file
license_gate.txt records whether an SPDX identifier search succeeded, which is a gate and not a full legal review.
Build the bundle with a script that can be rerun
The following shell script is an unexecuted proposal, not a recorded run from this account, and paths must be adapted. It writes the base commit, copies one fixture, captures status streams, probes for an SPDX line, and hashes the packet. A contributor should replace the sample CLI invocation with the command already documented by the upstream project. The hash file is the handle a later review uses to prove the packet did not drift between machines.
#!/usr/bin/env bash
# Proposal: unexecuted example. Adapt paths before use.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
SHA="$(git rev-parse HEAD)"
OUT="${ROOT}/.review-bundle"
mkdir -p "${OUT}/fixture"
echo "${SHA}" > "${OUT}/base_sha.txt"
cp "${ROOT}/repro/input.txt" "${OUT}/fixture/input.txt"
sha256sum "${OUT}/fixture/input.txt" > "${OUT}/fixture.sha256"
set +e
python3 -m project.cli --input "${OUT}/fixture/input.txt" \
> "${OUT}/stdout.txt" 2> "${OUT}/stderr.txt"
STATUS=$?
set -e
echo "${STATUS}" > "${OUT}/exit_status.txt"
if git grep -n "SPDX-License-Identifier" -- '*.py' > "${OUT}/spdx.txt"; then
echo "spdx-present" > "${OUT}/license_gate.txt"
else
echo "spdx-missing" > "${OUT}/license_gate.txt"
fi
find "${OUT}" -type f ! -name 'packet.sha256' -print0 \
| sort -z | xargs -0 sha256sum > "${OUT}/packet.sha256"
Running the script on a dirty worktree will mix unrelated edits into the story the packet tells the reviewer. A clean detached checkout of the recorded base commit is the honest place to capture the failing status. If that status is already zero, the contributor stops and does not ask a model to invent a bug. That stop is the point of the bundle, and it saves the maintainer from reviewing a speculative patch.
Climb three rungs before any model comment
Rung one runs the failing command on a clean checkout of the recorded base commit and expects the frozen exit status. Rung two applies the patch on that same commit and expects the command to exit zero. Rung three runs the repository test command, not a new test invented for the patch alone. A model should not be asked to invent those commands after the fact, because that invites a moving target.
Repeat the rungs on a second checkout
git fetch origin
git switch --detach "$(cat .review-bundle/base_sha.txt)"
bash scripts/oss-review-bundle.sh
test "$(cat .review-bundle/exit_status.txt)" = "1"
git apply --check patches/fix.diff
git apply patches/fix.diff
# Proposal: after the patch, rerun the documented command and expect status 0.
The first test line fails the workflow when the recorded status is not reproduced, which is the desired alarm. The apply check catches a patch that does not fit the recorded base commit, before anyone discusses coding style. After the patch applies, the same command should exit zero, and the repository test command should still pass. A model review that arrives before these three results exist is commentary, not evidence a maintainer can trust.
Minimize the fixture without deleting the failure
A long log invites a model to summarize noise and then miss the line that actually fails the command. The contributor should cut the fixture in halves, rerun the failing command, and keep the half that still fails. Each cut is recorded by appending the new fixture hash to a short minimization log stored inside the bundle. When a cut makes the status become zero, that cut is discarded and the previous fixture remains the public input.
Steps that keep the failure alive
- Copy the original fixture into the bundle and record its hash, its line count, and the failing exit status.
- Split that fixture into two halves and rerun the documented failing command against each half on the recorded commit.
- Keep the smallest half that still returns the frozen status, and append that hash to the minimization log.
- Stop when neither half fails, restore the previous fixture, and treat that file as the public input for review.
The log format below is a proposal with placeholder hashes, not measured output from a repository. A real run replaces those tokens with sha256sum output and the exit status just observed. The discarded row stays in the log so a reviewer can see which cut destroyed the failure. That trail matters more than a shorter file that no longer fails.
step=0 lines=48 sha256=placeholder-original status=1 kept
step=1 lines=24 sha256=placeholder-half status=1 kept
step=2 lines=12 sha256=placeholder-quarter status=0 discarded
Score review comments by citation
A model review earns trust only when each remark names a file that already sits inside the frozen bundle. A comment that calls the patch correct without naming exit_status.txt or fixture.sha256 scores zero for evidence. A comment that cites a packet file and quotes a line can be checked by a human in one pass. This scoring habit resists fluent agreement that replaces a failed reproduction on the recorded base commit itself.
The table below is a proposal for sorting comments, not a measured study from this account's history. Strong rows still require a human to open the cited file before the patch is allowed to move. Weak rows are corrections to the note, not reasons to merge the patch into the default branch. Empty rows are discarded even when the wording sounds confident, polite, and ready for a maintainer.
Model comment patternPacket citationHuman actionFluent praise with no filenameNoneDiscard the commentCites
exit_status.txt but quotes the wrong digitWeakOpen the file and correct the noteCites
fixture.sha256 and the minimization logStrongCompare hashes, then read the diffSuggests a test command absent from the repo docsOut of packetIgnore until a human adds that command
# Proposal: unexecuted. Scores whether review lines cite packet files.
from pathlib import Path
PACKET = {
"base_sha.txt",
"exit_status.txt",
"fixture.sha256",
"license_gate.txt",
"packet.sha256",
}
text = Path("model-review.md").read_text(encoding="utf-8")
cited = {name for name in PACKET if name in text}
missing = PACKET - cited
print(f"cited={sorted(cited)}")
print(f"missing={sorted(missing)}")
if not cited:
raise SystemExit("review cited no packet file")
The checker is also an unexecuted proposal, and it only tests whether filenames appear in the review text. It cannot tell whether the model understood the exit status, so a human still opens every cited file. If the checker exits because no packet file was named, the review is rejected without a second pleading prompt. A second prompt that begs for agreement would only recreate the original failure mode this bundle is meant to stop.
Use free model access only after the hash is frozen
MonkeyCode is relevant here only as an optional place to run the citation pass and to repeat the bundle script. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied claim used here is that free model access and a free server option currently exist. The operator also describes the project as open source, which readers should confirm in the repository before they depend on it.
This article does not state model names, token quotas, hardware sizes, or how long that access lasts. Readers should confirm current terms on the project page before relying on either option for a real review. A missing or changed offer does not break the ladder, because the packet can be reviewed by a person alone. Hosted help is a convenience after the hash exists, not a requirement for the reproduction or the patch.
A free model pass is useful after the packet hash is frozen, because the model then has a closed set of files to cite. A free server rerun is useful when the contributor's laptop has extra packages that the upstream image does not install. Both steps stay optional, and the ladder still works with a local model or with a second container on the same desk. The packet hash, not the model's tone, remains the acceptance criterion for moving the patch toward a human reviewer.
The prompt should ask for missing packet fields, mismatched hashes, and a license gate that printed spdx-missing. It should not ask for a verdict that the patch is ready to merge, because that verdict is a human decision. Output belongs in model-review.md beside the packet, so the filename checker can read it without extra parsing. The contributor then keeps only sentences that name a packet file before the maintainer sees the note.
The contributor deletes any sentence that does not name a packet file before the maintainer sees the note. That deletion is a human edit, and the pull request should say the model text was filtered by the checker. A maintainer can then review the patch, the packet, and a short list of cited gaps without rereading praise. The model does not appear as an author of the commit, because authorship stays with the person who can answer questions.
Limitations that should stay visible
This workflow does not prove that a patch is correct, and it does not replace the maintainer's review or the project's legal process. A minimized fixture can drop the real bug if the contributor cuts the input until the failure disappears. License header searches miss files that use a different SPDX line, or a license that lives only in a top-level file. A model can still cite the right filename and misread the exit status, so a human must open the cited file.
The free server option, where it is available, does not guarantee the same packages as the project's continuous integration image. A rerun that passes on a generic machine can still fail inside the upstream workflow file on the next push. Contributors should treat a green second machine as a portability hint, not as a substitute for the project's own checks. Quotas, model names, and how long free access lasts sit outside this article and must be read from current terms.
Who should skip the bundle
Skip this bundle when the repository forbids attached fixtures, or when the bug depends on private data that cannot be reduced. Skip it when the change is a documentation typo with no failing command, because a fake exit status would only add noise. Skip it when the contributor cannot read the test output well enough to reject a confident but wrong model comment. In those cases a short issue comment and a human review remain the honest path, and a model pass would not help.
What the maintainer should be able to replay
A maintainer should be able to detach at the recorded commit, rerun the bundle script, and see the same fixture hash. The patch should flip the frozen status, and the citation checker should show at least one packet file in the review note. If either hash differs, the conversation stops until the contributor publishes a new packet rather than explaining the drift in chat. That replay is the whole method, and it remains useful even if no hosted model or free server is used.
Readers who still want a clean second machine for that replay can check whether MonkeyCode's free server option is open. They should require the same packet hash on that machine before they trust the rerun at all. The free model access, if currently offered, fits only the citation pass after the hash file already exists. Neither option is a merge approval, and the project page is the place to confirm what is available that day.#opensource #testing #git #ai #software #coding #development #engineering #inclusive #community
A Citation Check for OSS Patches Starts With a Minimized Public Fixture
