Evidence · last run October 19, 2026
Four rounds, eleven public repos, about 28,000 merged pull requests, and the claims written down before each round ran. All of it is below, including what failed.
01 · The scoreboard
Round 1 used the original rules on the repos they were tuned on (Sentry, Grafana) and on one they never saw (Supabase). Round 2 used revised rules on three fresh repos picked for having the most reverts (next.js, Airflow, Kibana). Round 3 fitted rules to each of those repos’ own history, month by month. A claim holds only if its 95% range clears the line; “not shown” means the data can’t tell, not that the opposite is true.
* Round 3, claim 1: the fitted pen did beat the same share of largest PRs, but mainly because on these repos the largest PRs were reverted less often than average. Round 2, claim 4: holds pooled, but Kibana’s easy pen had the highest revert rate of its three pens, and was slightly above the smallest PRs.
02 · Round 4
Rounds 1–3 gave every method the same number of pull requests to flag. Review effort is closer to lines read, so round 4 gave every method the same lines as the careful pen. We chose this measure after the earlier rounds showed no win, so we wrote the claims down first, kept the rules unchanged, and used three repos never touched before: Elasticsearch, VS Code and Home Assistant (205 later-reverted PRs).
* Claims 1 and 2 pass, but they aren’t a win worth claiming: with a lines budget, largest-first spends everything on a handful of huge PRs (3 in Elasticsearch). The honest reading is claim 3 and smallest-first. An independent recomputation matched every figure.
| PR size | share of PRs | share of lines | reverted within 30 days | reverts per 10,000 lines |
|---|---|---|---|---|
| under 10 lines | 25.8% | 0.3% | 0.53% | 11.80 |
| 10–49 | 23.5% | 1.4% | 0.70% | 2.78 |
| 50–199 | 23.5% | 6.2% | 0.46% | 0.43 |
| 200–999 | 20.8% | 22.7% | 0.85% | 0.19 |
| 1,000+ | 6.3% | 69.4% | 0.68% | 0.02 |
A pull request’s chance of being reverted barely changes with size, while lines change a thousandfold. Careful leans toward big PRs: 11–19% of PRs but 32–49% of the lines. So per line read, it finds less than picking at random.
If review time tracks lines read, a quick look at every small PR finds more of what later gets reverted than a deep look at a few big ones. That fits the one claim that has held every round: the easy pen is as safe as the smallest PRs, no more and no less.
Reverts are one kind of trouble, and big PRs are harder to revert, so they may get fixed forward instead. Counting only reverts may understate trouble in big PRs. Follow-up fixes were too sparse in these repos to check.
03 · Habits
We asked whether three habits visible in git history go with fewer reverts: no shortcuts (no silenced warnings, skipped tests and the like), tests changed with the code, and linking an issue. Claims were written first; the counted repos were PyTorch, Ray and Playwright (1,699 reverted pull requests among source changes), with six more repos from earlier rounds reported but not counted. Size was held fixed.
* An independent recomputation matched the pre-registered numbers, then found two flaws: 175 PyTorch relands were counted as reverted, and most “issue links” in PyTorch were references to other pull requests in a stack. The corrected numbers are in the second column. PyTorch supplies 1,629 of the 1,699 reverts, so it decides the pooled result; without it, no habit reaches a verdict.
PyTorch’s merge bot reverts fast, mostly for breaking tests or internal builds. A pull request that changes tests is likelier to break tests. That says what PyTorch reverts for, not that writing tests is a bad habit.
No cross-team “practices score”: the data doesn’t support one. Cowboy’s Habits page shows each repo its own shares, and only says a habit “holds here” when that repo’s history shows it.
Reviews, approvals, checks before merge and disclosed agent help aren’t in git history, so they weren’t tested. A live Cowboy install reads them from the git host.
04 · Rounds 2 and 3, in numbers
151 later-reverted PRs across next.js, Airflow and Kibana, six times as many as the first held-out test. Rates are weighted back to each repo’s real mix of PRs.
| repo | easy | normal | careful | all PRs | sent to careful | careful vs average | fitted pen vs average |
|---|---|---|---|---|---|---|---|
| vercel/next.js 46 reverted PRs | 0.94% | 1.34% | 1.71% | 1.28% | 20% | 1.33× | 0.87× (11% of PRs) |
| apache/airflow 32 reverted PRs | 0.29% | 0.50% | 0.72% | 0.45% | 14% | 1.60× | 3.53× (6% of PRs) |
| elastic/kibana 62 reverted PRs | 0.48% | 0.27% | 0.33% | 0.34% | 29% | 0.95× | 1.29× (15% of PRs) |
Careful PRs were reverted 1.08× as often as average (range 0.78× to 1.39×). The revisions made the pen smaller in two of three repos, but not better: the original rules did about the same here.
A pen half the size of the defaults’ (12% of PRs vs 24%) with a higher hit rate: 1.37× average vs 1.10×. But the range (0.87× to 1.93×) includes chance, so it can’t be claimed. And months without enough history fell back to the defaults, which supplied 8 of its 17 catches.
In Kibana it kept picking the CI configuration folder, and that carried over: those PRs were reverted about 3.1× as often in the following months (5 reverts, so still thin). In next.js it picked single files with three to five reverts each, and last quarter’s trouble files weren’t next month’s.
05 · Round 1, claim by claim
Each bar is the range the result could plausibly fall in (a 95% bootstrap range, 2,000 resamples). If the bar crosses the dashed line, the claim isn’t shown.
Does the careful pen hold more of the later-reverted PRs than the same share of the largest PRs?
Same, against the PRs that touch the most files.
Are careful PRs reverted more often than the repo’s PRs overall?
Is the easy pen reverted no more often than the same share of the smallest PRs?
06 · Round 1: where the reverts landed
Give every method the same budget: the share of PRs Cowboy put in careful. Then count how many of the PRs that were later reverted each one caught.
Each method flags the same share of PRs as the careful pen. Careful leads size a little and ties spread; neither gap is outside the noise.
Only 25 reverted PRs, so every bar here is rough. Careful does no better than the largest PRs.
07 · Round 1: per pen, per repo
Weighted back to each repo’s real mix of PRs. Reverts are rare everywhere, so small differences matter. Read the PostHog row as a warning: its careful pen took 38% of all PRs and was reverted less often than average.
| repo | easy | normal | careful | all PRs | sent to careful | reverted PRs in sample |
|---|---|---|---|---|---|---|
| getsentry/sentry tuned on |
0.81% | 1.23% | 1.42% | 1.18% | 29% | 174 |
| grafana/grafana tuned on |
0.34% | 0.36% | 0.69% | 0.47% | 36% | 40 |
| n8n-io/n8n tuned on · too few reverts to count |
0.00% | 0.02% | 0.00% | 0.01% | 20% | 1 |
| supabase/supabase held out |
0.48% | 0.46% | 0.58% | 0.48% | 10% | 25 |
| PostHog/posthog held out · too few reverts to count |
0.05% | 0.13% | 0.08% | 0.10% | 38% | 18 |
A pen’s rate shows what happened, not that the pen caused it. No team was using Cowboy on these PRs.
08 · How the tests work
The four claims, the alternatives and the pass mark were committed to the repo before the comparison ran. The commit time is the proof. The plan also said: if a claim fails, this page says so.
Merged PRs from November 2025 to September 2026 in five public repos, each sorted by Cowboy’s current rules. “Had trouble” means reverted within 30 days, found by Cowboy’s own after-merge engine from the repo’s history. A repo needs 20 reverted PRs to count.
The parser. The first run on Sentry, Grafana and n8n used Cowboy’s older import finder, not the parser it ships. Re-run with the shipped parser, “careful beats chance” no longer holds there either. This page shows the re-run. The sample. The plan called the sample random. It wasn’t: every reverted PR was kept, plus a random slice of the rest. That made reverts look four to five times more common than they are. The first run, unweighted, said claim 1 held. Weighted back to each repo’s real mix, it doesn’t. This page reports the weighted run; the unweighted one is in the results file.
09 · Can you trust the measuring?
of 250,319 JavaScript and TypeScript imports in Sentry, Grafana and n8n found, and wherever the TypeScript compiler could resolve an import, 99.9% or more went to the same file it picks. Every file TypeScript says has 20+ importers, Cowboy caught. Python imports matched Python’s own parser.
reverted PRs found by both Cowboy’s engine and an earlier, separate count across the five repos. They disagreed on 17: 11 only the earlier count found, 6 only the engine found. Neither is a hand-checked truth.
The sorting is fixed rules with sorted iteration: the same PR and the same code give the same pen and the same reasons, every run.
10 · What this doesn’t show
The rules were adjusted while looking at Sentry, Grafana and n8n. The held-out test is the fairer one, and it’s the weaker result.
Broken main, follow-up fixes and incidents need data public repos don’t keep. Reverts are the clearest kind of trouble, and also the rarest.
Nobody reviewed differently because of a pen. This checks whether the pens point at risky PRs, not whether a team using Cowboy ships fewer of them.
Supabase has 25 reverted PRs and PostHog 18. That’s enough to see a big effect, not a small one.
11 · What happens next
Three rounds say no fixed rule set, and no simple fit, reliably finds risky PRs across repos. What did work every time is the part that measures: finding what got reverted, tying it back to the PR, and checking any rule against it. That’s what Cowboy will lead with.
Reverts are rare: a few a month even in busy repos, too few to judge a rule quickly. Counting main breaking after a merge and confirmed follow-up fixes gives several times more events to learn from.
Cowboy can export a counts-only summary (no names, paths or code). Teams that share theirs add evidence from live use, which no replay can.
12 · Run it yourself
The evidence folder has the claims, the results and the comparison script. Collect a repo’s PRs, sort them, replay the after-merge engine, then compare:
git clone --filter=blob:none --bare https://github.com/OWNER/REPO repo.git python3 scripts/eval/collect_sort.py repo.git OWNER/REPO full.jsonl checkout 2> collect.log python3 scripts/eval/checkout_src.py repo.git COMMIT checkout # COMMIT: last line of collect.log go run ./cmd/cowboy-sortlab -in full.jsonl -checkout OWNER/REPO=checkout -out sorted.jsonl go run ./cmd/cowboy-replay -sorted sorted.jsonl -patches full.jsonl -collectlog collect.log -git OWNER/REPO=repo.git -out replay python3 scripts/eval/evidence/compare.py sorted.jsonl replay collect.log