01
#13 · more-itertools all_equal edge case
GitHubReference label SAFE · Model signal LOWER · Ranked a safe change as lower review priority.
ML evaluation · Developer tool
PatchCheck
Coding-agent patches can look plausible while missing requirements or breaking adjacent code. PatchCheck tests whether a small verifier can help reviewers decide which PRs to inspect first.
- Role
- Independent project · Dataset, QLoRA training, evaluation, and GitHub Action
- Technologies
- Qwen2.5-Coder-7B · QLoRA · PyTorch · Modal · GitHub Actions
Actual product
See it in action
These five PRs demonstrate that the GitHub Action and Modal integration work in a real review flow. They are deployment examples, not a separate benchmark of generalization performance.
02
#16 · moto ECR field-allowance regression
GitHubReference label UNSAFE · Model signal HIGH · Ranked a risky change for immediate review.
03
#17 · flake8-comprehensions C416 false positive
GitHubReference label SAFE · Model signal LOWER · Ranked a safe change as lower review priority.
Additional examples
- Training patches with execution labels
- 11,742
- Familiar held-out ROC-AUC
- 0.8200
- New-repository OOD ROC-AUC
- 0.7479
- New-agent OOD ROC-AUC
- 0.6751
01
Context
A coding-agent patch can look reasonable while solving only part of an issue or breaking an adjacent path. Instead of calling a larger LLM again, I tested whether a small model that sees only the issue and candidate diff could provide a useful review-order signal.
A familiar held-out score does not describe a production review tool. I evaluated new repositories, unseen agent families, size-matched SAFE and UNSAFE pairs, and external data with the same protocol. The model does not decide correctness or automatically approve PRs.
02
Engineering
A QLoRA adapter is trained on Qwen2.5-Coder-7B-Instruct with the issue description and candidate diff as its only input. Zero-shot predictions from the same frozen 7B model provide the control for measuring the effect of fine-tuning.
The verifier converts the sequence-likelihood difference between PASS and REVIEW into a risk percentile. A separate CPU analyzer checks API, authentication, dependency, exception, secret-like, and test-coverage changes. GitHub Actions runs the CPU checks, while a maintainer-gated Modal endpoint performs GPU inference.
My contribution
- Built the training population, fixed evaluation splits, QLoRA training, and inference path.
- Designed familiar, repository OOD, agent-family OOD, same-task hard-match, and external-review evaluations.
- Implemented base/head SHA diff collection and suppressed model scores when the diff was missing or too large.
Key technical decisions
Use the same base model as the control
Prompt, tokenizer, and score definition remain fixed. This isolates the change introduced by fine-tuning instead of comparing unrelated models.
Control the patch-size shortcut
The 1,000 same-task SAFE and UNSAFE pairs have similar changed-file counts and patch sizes. The score cannot rely only on treating larger patches as more dangerous.
Display model and static-analysis signals separately
A learned ranking signal and a deterministic code finding carry different meanings. They remain separate in the review UI instead of being presented as one correctness probability.
03
Validation
ROC-AUC decreased from familiar data to new repositories and new agents. Shortcut-controlled hard cases measured 0.5972 and the external set measured 0.6540. Every split exceeded the frozen zero-shot control, but the external unsafe PR-AUC confidence interval included zero, so the improvement is not presented as conclusive.
Uncertainty uses 10,000 paired nonparametric bootstrap samples over frozen seed-17 predictions. It describes evaluation-sample uncertainty, not training-seed variance.
Verified results
- The hard-case verifier exceeded the structural-shortcut baseline of 0.4974 and metadata baseline of 0.5011.
- The product reports LOWER, ELEVATED, and HIGH review priority against a frozen reference distribution, not a correctness probability.
Failures and changes
- Combining model and static-analysis scores reduced shortcut-controlled unsafe PR-AUC from 0.5900 to 0.5150. The combined score was removed, and the product now shows both signals separately.
Next steps
- The model sees only the issue and diff, so it cannot observe repository execution or hidden-test results. CI and static-analysis evidence should be attached separately so reviewers can compare learned risk with actual failures.
- External-review gains are small, and some confidence intervals include zero. The tool should remain a review-priority signal until larger repository and agent evaluation sets support a new calibration.