Docs

Overview

Cowboy sorts every pull request into a pen, and tags the ones your AI coding agents opened:

• easy: small, low-risk changes you can skim.
• normal: a regular review.
• careful: risky changes a code owner should look at.

It picks the pen with fixed rules, so the same pull request always lands in the same pen, and it writes down why. It also flags doubles (two agents doing the same work) and strays (an agent that went quiet after you asked for changes).

It looks back at what merged, to show which rules and habits actually went with less trouble in your repo, and it can turn your rules into instructions your coding agents follow.

Cowboy runs on your own server. It only reads your pull requests and adds labels, one comment and review requests. It never merges, closes or reviews code, and your code isn't sent anywhere except an optional model server you run yourself.

Quick start

The demo runs Cowboy on a made-up sample repo, with nothing to set up and no account needed.

macOS or Linux

curl -fsSL https://github.com/sgracia13/Cowboy/releases/latest/download/install.sh | sh
cowboy demo

The install script downloads the right file for your computer, checks it against the published checksums and puts it in /usr/local/bin, or ~/.local/bin if that isn't writable. The macOS builds aren't signed yet; the script clears the download quarantine for you.

Windows

Download cowboy-windows-amd64.exe from the releases page and run cowboy-windows-amd64.exe demo.

With Go

go run github.com/sgracia13/Cowboy/cmd/cowboy@latest demo

What you'll see

Open http://localhost:8080 straight away. The sample pull requests arrive one every 4 seconds and drop into their pens without a refresh. One webhook is skipped on purpose, so you can watch the catch-up sweep find that pull request a few seconds later. Click any card to see its reasons and the comment Cowboy would post.

Want everything at once? cowboy demo --drip 0. Just a table in the terminal? cowboy demo --once.

Then open outcomes: a replay of real vercel/next.js history (about 2,500 merged pull requests, author names removed), so you can see what Cowboy finds after merge and how the pens compare with simple alternatives there.

Try it on your repo

cowboy

With no arguments, Cowboy starts on this machine only and opens your browser on a setup page:

1. Connect GitHub. One click to use your GitHub CLI (gh) login if it's signed in, or paste a token. The open github's token page button opens GitHub with the name and read-only permissions filled in; pick the repos under Repository access and press Generate.
2. Pick repos from the list it can see.
3. Choose what it does: just look (the default: sort pull requests and check past ones, writing nothing to GitHub) or label pull requests.
4. Save. Cowboy sorts the open pull requests, rates the ones merged in the last 90 days, and takes you to the outcomes page while it works.

When the pens look sensible, press start labelling on the setup page. Cowboy first checks the login can write labels on every repo, by adding the cowboy:easy label, and tells you if it can't. The login, repos and mode are saved, so next time it's just cowboy. Sign out or swap the token on the same page.

On a laptop GitHub can't send webhooks to Cowboy, so it checks for new pull requests every 5 minutes instead. For a server your team shares, see the next section.

Run it for your team

On GitHub, Cowboy can set itself up: it creates a private GitHub App for your server, and you pick the repos. On Forgejo and Gitea, or if you'd rather use a token, follow the steps further down.

Connect GitHub with one click preview

docker run -d --name cowboy -p 8080:8080 -v cowboy-data:/data \
  -e UI_PASSWORD=a-dashboard-password \
  -e PUBLIC_URL=https://cowboy.your-domain.com \
  ghcr.io/sgracia13/cowboy:latest

1. Open the dashboard and sign in. A banner says Cowboy isn't connected yet.
2. Press create the github app. To create it under an organization, type the organization's name first.
3. GitHub shows the app it's about to create. Confirm it (you can change the name there).
4. Pick the repos Cowboy should sort, then press Install.
5. You land back in the corral, and the open pull requests appear as they're sorted.

That's all: no token, no webhook and no secret to copy. Repos you add or remove on GitHub later are picked up automatically.

• What it asks for: pull requests (read and write), issues (read and write, for labels and the comment), contents (read, for cowboy.json and CODEOWNERS), checks and metadata (read).
• How it signs in: with one-hour tokens per installation, not a long-lived personal token. The app's private key and webhook secret are saved in github-app.json next to the data file, readable only by Cowboy's user. Back it up with the data volume.
• Webhooks: with a PUBLIC_URL GitHub can reach over HTTPS, they're turned on and pull requests are sorted the moment they change. On your own machine or a private address, the app is created with webhooks off and the catch-up sweep checks every 5 minutes instead.
• Trying it on your laptop: run cowboy serve and open http://localhost:8080. No password or PUBLIC_URL is needed on your own machine.
• GitHub Enterprise: set FORGE_URL to your API address, like https://ghe.your-co.com/api/v3.
• An app you made yourself: set GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY_FILE and WEBHOOK_SECRET.

This setup is new. It's tested end to end against a stand-in for GitHub's API, but hasn't yet been run against github.com.

Or: with a token

You need a server your git host can reach over HTTPS, a token Cowboy can use, and a webhook. This uses GitHub; for Forgejo and Gitea see the next section.

1. Create a token

   In GitHub, go to Settings › Developer settings › Fine-grained tokens, choose the repos Cowboy should sort, and give it these permissions:

   Permission	Access	Used for
   Pull requests	Read and write	reading pull requests, diffs and reviews; asking code owners for a review
   Issues	Read and write	labels and the comment (GitHub handles these for pull requests through issues)
   Contents	Read	reading cowboy.json and CODEOWNERS from the default branch
   Checks	Read	knowing whether checks pass
   Metadata	Read	required by GitHub

   A classic token with the repo scope also works.

2. Start Cowboy

   With Docker:

   docker run -d --name cowboy -p 8080:8080 -v cowboy-data:/data \
     -e FORGE=github \
     -e FORGE_TOKEN=github_pat_... \
     -e WEBHOOK_SECRET=a-long-random-string \
     -e UI_PASSWORD=a-dashboard-password \
     -e REPOS=your-org/your-repo \
     ghcr.io/sgracia13/cowboy:latest

   Or with the binary, using the same settings as environment variables:

   export FORGE=github FORGE_TOKEN=github_pat_... WEBHOOK_SECRET=... UI_PASSWORD=...
   export REPOS=your-org/your-repo DATA=/var/lib/cowboy/cowboy.json
   cowboy serve

   When it starts, the log says whether the token can read each repo in REPOS, then Cowboy sorts the pull requests that are already open.

   Try a dry run first. Add DRY_RUN=1 and Cowboy sorts everything and shows it on the dashboard without writing anything to GitHub. Remove it when you're happy.

3. Add the webhook

   In the repo (or the whole organization), go to Settings › Webhooks › Add webhook:

   Field	Value
   Payload URL	https://cowboy.your-domain.com/webhook/github
   Content type	application/json (the default form type won't work)
   Secret	the same value as WEBHOOK_SECRET
   Events	Let me select: Pull requests, Check suites and Check runs

   Webhooks are how Cowboy hears about pull requests the moment they change. If one is lost, the catch-up sweep picks the pull request up within 5 minutes.

4. Open the dashboard

   Go to your Cowboy address and sign in with UI_PASSWORD. New pull requests appear in their pens as they're sorted.

Put it behind HTTPS

GitHub needs an HTTPS address for webhooks. The simplest way is a reverse proxy such as Caddy, which gets a certificate for you:

# Caddyfile
cowboy.your-domain.com {
  reverse_proxy 127.0.0.1:8080
}

Blast radius

To know how many files depend on a changed file, Cowboy builds an import graph of the repo. It does this by itself: there's nothing to set up.

• No local copy needed. Cowboy downloads the default branch as a compressed archive from your git host, through the same connection it already uses (the GitHub App's contents: read covers it). It reads the imports as the archive streams in and keeps only the list of who imports what. No code is written to disk.
• Kept current. Every hour it checks whether the default branch moved, and downloads again only if it did. Pull requests keep being sorted with the last graph while a new one builds.
• Real parsing. Go is read with Go's own parser. JavaScript, TypeScript and Python are read by scanners that handle imports split over many lines and skip comments and strings. Imports are matched to files using go.mod, tsconfig paths and baseUrl, workspace packages in monorepos, and Python package folders like src/.
• Checked against the real tools. On n8n, Grafana and Sentry it found 100% of the imports the TypeScript compiler and Python's own parser find, and caught every file that 20 or more files import. See the numbers.
• Size. Archives over 300 MB are skipped (set BLAST_RADIUS_MAX_MB to change it). n8n, Grafana and Sentry are 50 to 55 MB and take 3 to 7 seconds. Files over 2 MB, like bundles, aren't read.

The rules page on the dashboard shows how blast radius is working for each repo. To read a local copy instead, set REPO_PATHS; to turn downloads off, set BLAST_RADIUS=off.

# optional: use a checkout you keep up to date yourself
export REPO_PATHS=your-org/your-repo=/srv/cowboy/your-repo

Run it as a service

For the binary on Linux, a minimal systemd unit:

# /etc/systemd/system/cowboy.service
[Unit]
Description=cowboy
After=network-online.target

[Service]
EnvironmentFile=/etc/cowboy.env
ExecStart=/usr/local/bin/cowboy serve
Restart=on-failure
User=cowboy

[Install]
WantedBy=multi-user.target

On stop, Cowboy finishes the work already queued before it exits.

GitLab preview

GitLab.com and self-managed GitLab both work. Merge requests are sorted, labelled and commented on like pull requests, and after-merge tracking reads the default branch, releases and pipelines.

FORGE=gitlab FORGE_URL=https://gitlab.com FORGE_TOKEN=glpat-... WEBHOOK_SECRET=... \
REPOS=your-group/your-project cowboy serve

• Token: a personal, project or group access token with the api scope (or read_api with DRY_RUN=1 to only look). Project and group tokens show up as a bot user, which keeps Cowboy's labels apart from people's.
• Repos: the project's full path, subgroups included: acme/platform/web.
• Webhook: project → Settings → Webhooks, URL https://cowboy.your-domain.com/webhook/gitlab, secret token = WEBHOOK_SECRET, triggers: merge request events and pipeline events.
• Reverts: GitLab's Revert button, git revert and revert merge requests are all traced back to the merge request they undid.

Limits: strays need a GitLab version that records "requested changes" on reviewers. If GitLab refuses several reviewers (a paid GitLab feature), Cowboy asks the first code owner only. Tested against a stand-in for GitLab's API, not yet against gitlab.com itself.

Forgejo and Gitea preview

Cowboy works the same way on Forgejo and Gitea. This support is covered by tests but hasn't been run against a live server yet, so please report anything odd.

• Set FORGE=forgejo (or gitea) and FORGE_URL to your server, e.g. https://git.your-domain.com.
• Create an access token with read and write access to repository and issue.
• Add a webhook with the URL https://cowboy.your-domain.com/webhook/forgejo (or /webhook/gitea), content type application/json, your secret, and pull request events.

One difference: on Forgejo and Gitea, changing a cowboy: label by hand isn't read back as a correction yet. Use the move buttons on the dashboard instead.

Settings

Cowboy is configured with environment variables. On a server you need a way to sign in (UI_PASSWORD or AUTH_PROXY_HEADER). With the GitHub App, add PUBLIC_URL; with a token, add FORGE_TOKEN and WEBHOOK_SECRET. Everything else has a default.

If the dashboard would be reachable from other machines and no login is set, Cowboy refuses to start and tells you what to set.

How sorting works

Every pull request goes through the same rules, whoever wrote it: the rules look at the code, not the author. Agent pull requests also get an agent tag (see Recognizing agents).

Pull requests from people stay quiet

A pull request with no agent mark gets a pen label like cowboy:easy and nothing else: no agent label, never a double or stray label, and a comment only when it lands in careful (where a heads-up is worth it). Change that with "people": {"comment": "always"} or "never" in cowboy.json, or sort agent pull requests only with "detect": {"sort_all": false}.

Why sort everything? Agent work that nobody marked can't be spotted reliably, so sorting every pull request is how nothing risky goes unsorted.

Careful, if any of these is true

• It touches a sensitive path. By default: auth/**, billing/**, payments/**, migrations/**, .github/workflows/** and security/**, wherever they sit in the repo.
• It changes one of the repo's most-imported files: among the 1% of files the most others import, and imported by at least 20 (blast radius). In a big repo, 20 importers is ordinary, so the bar rises with the repo.
• It adds a catch-all that hides errors: a broad catch whose body does nothing with the error, like an empty catch {}, except Exception: pass or a return null, or a bare Python except:. A broad catch that logs or re-raises doesn't count.
• It loosens CI: || true, continue-on-error, removed test steps and similar.
• It loses test assertions across the whole pull request. Assertions moved to another test file don't count, and neither do pull requests that remove a feature along with its tests.
• Its checks are failing.

Easy, if it isn't careful and

• it changes 50 lines or fewer, in 2 folders or fewer, and 3 or fewer files import what it changed, and its checks aren't failing; or
• it changes no source or generated code at all (only tests, docs or config) and 600 lines or fewer (12 times easy.max_lines).

Normal

Everything else.

Worth a look

Some things are shown on the pull request as notes but never change the pen: warning silencers (like eslint-disable or # type: ignore), newly skipped tests, comments that admit special-casing for tests, and pull requests with 400 or more lines of source, which get a "too big to review well" note.

Doubles and strays

• A double is a newer agent pull request that links the same issue (Fixes #12) as an older open one, or changes 80% or more of the same files. It gets the cowboy:double label.
• A stray is an agent pull request with no new commits 3 days after changes were requested. It gets cowboy:stray; after 7 days Cowboy suggests closing it. Strays are checked every hour, or when you press check for strays.

How well the rules work

Not proven yet. On about 28,000 merged pull requests from eleven public repos, tested with the claims written down first, the careful pen didn't beat “largest first” or chance at finding pull requests that were later reverted. The easy pen mostly held up: it was no riskier than the same share of the smallest pull requests. So read careful as “a closer look might be worth it”, not “this will break”, and check the pens on your own repo with the comparison on the outcomes page. The evidence page has every test.

Per-repo rules

Put a cowboy.json (or .cowboy.json) on the repo's default branch to change the rules for that repo. Cowboy always reads it from the default branch, so a pull request can't loosen its own rules. Anything you leave out keeps its default.

{
  "agents": { "build-bot": "our-agent" },
  "careful": {
    "paths": ["auth/**", "billing/**", "infra/**"],
    "min_fan_in": 20,
    "top_fan_in_percent": 1,
    "shortcuts": ["tests_removed", "catch_all", "ci_loosened"]
  },
  "easy": { "max_lines": 50, "max_dirs": 2, "max_fan_in": 3 },
  "notes": { "big_source_lines": 400 },
  "strays": { "nudge_after_days": 3, "suggest_archive_after_days": 7 },
  "fan_in_ignore": ["src/utils/index.ts"],
  "people": { "comment": "careful" },
  "detect": { "enabled": false, "sort_all": true }
}

The rules page on the dashboard shows the rules each repo is actually using.

Code owners

For careful pull requests, Cowboy asks the matching people in CODEOWNERS (in .github/, the repo root, docs/ or .gitea/) for a review. It skips the pull request's own author, and in this preview it skips teams like @org/team.

Recognizing agents

Cowboy labels a pull request as agent work when it finds one of these:

• a co-author line in a commit, like Co-authored-by: Claude <noreply@anthropic.com>;
• a bot account as the author, like copilot-swe-agent[bot];
• a signature in the pull request description, like "Generated with Claude Code".

Built in: Claude Code, GitHub Copilot, Codex, Devin, Cursor and Jules. These are only recognized when the agent leaves one of those marks; an agent run on someone's laptop that leaves no mark looks like a person. Add your own bots under agents in cowboy.json.

The label is agent:<name>, for example agent:claude-code.

Unmarked pull requests experimental

Cowboy can also score pull requests that carry no mark, comparing each one to that author's own history: commit timing, writing style and code shape. If even the low end of its estimate reaches 80, the pull request gets agent:likely and is treated like other agent work. Either way it's sorted into a pen.

• It's on by default in this preview. Turn it off with "detect": {"enabled": false}.
• Remove the agent:likely label, or press not an agent on the dashboard, and Cowboy treats the pull request as human work and asks for more evidence on that author from then on.
• "label_at" changes the threshold (default 80).
• Its per-author history is kept in baselines.json next to the data file.
• To see how it reasons, run cowboy demo --agent-check. The sample authors there are invented, so it shows the method, not how well it works on real repos.

The dashboard

• Corral: every open pull request in its pen, agent work tagged, with doubles and strays flagged at the top. It updates live; the yellow dot in the header means it's connected.
• Pull request page: click any card. Shows why it's in its pen, what Cowboy measured, local model notes if you have a model, and the comment it posted. Use the buttons to move it to another pen; Cowboy updates the label and counts the move in the tally.
• Tally: the last 7 days by agent, with pull requests from people as one row: how many pull requests, in which pens, doubles, strays, shortcuts and how often your team moved one. It can be sent to Slack or by email every week (below), and has a ready-to-paste version too.
• Rules: the rules each repo is using, and the model and agent check settings.
• After merge: what happened to merged pull requests, by pen and by rule (below).
• Habits: the habits each repo keeps and the agent guide (below).
• Activity: everything Cowboy did, newest first.

Get the tally every week

Cowboy can post the tally to a Slack channel, email it, or both, every Monday at 9:00 by default. Nothing is sent unless you set a destination.

# Slack: create an incoming webhook for the channel (Mattermost webhooks work too)
TALLY_SLACK_WEBHOOK=https://hooks.slack.com/services/...

# Email
TALLY_EMAIL_TO=team@your-co.com,lead@your-co.com
SMTP_HOST=smtp.your-co.com SMTP_USER=... SMTP_PASSWORD=... SMTP_FROM=cowboy@your-co.com

# Optional
TALLY_DAY=friday TALLY_HOUR=16 TZ=Europe/Berlin
PUBLIC_URL=https://cowboy.your-domain.com

The post covers the 7 days before it goes out: pull requests per agent and for people, by pen, with doubles, strays and how often your team moved one. In Slack it's a short summary and a table; by email it's a small html table with a plain-text copy.

The first post goes out at the next scheduled time. To try it straight away, run cowboy tally --send with the same settings. The tally page on the dashboard says where it's sent and when the next one goes out.

Cowboy remembers what it sent in tally-sent.json next to the data file, so a restart never posts twice. If a send fails, it tries that destination again every 30 minutes; a post more than a day late (the server was off) is skipped instead of sent stale.

On GitHub you can also move a pull request by changing its cowboy:easy, cowboy:normal or cowboy:careful label yourself; Cowboy remembers your choice.

Signing in

• Password: set UI_PASSWORD. Everyone shares one login, and moves are recorded as "team".
• Sign-in proxy: behind oauth2-proxy, Authelia, Cloudflare Access or similar, set AUTH_PROXY_HEADER to the header it sets. Moves are recorded under each person's name.

Dashboard buttons only accept requests from the dashboard itself, and the pages send strict browser security headers.

After merge

Cowboy follows every merged pull request for 30 days and ties what happened back to it. The outcomes page shows what went wrong, and whether the pens, or any simpler rule, would have pointed at it. It's on by default; set OUTCOMES=off to turn it off.

What counts as trouble

Anything that counts can be marked "not a problem". Cowboy also records who approved each pull request, which release it shipped in, and how much of its code is still there after 30 and 90 days. It measures that from fingerprints of a sample of the added lines; no code is kept.

The outcomes page

• Do the pens beat simple alternatives? See below.
• By pen: how often easy, normal and careful pull requests had trouble, with a range, so "2 of 10" doesn't look like a fact. Only pull requests merged at least 30 days ago are rated, so recent ones don't look cleaner than they are.
• Rule report card: for each careful rule, how often its pull requests had trouble, and whether careful pull requests approved before merge did better than ones that weren't.
• Suggested rule changes: only when the numbers clearly point somewhere, like a folder whose easy pull requests keep breaking. Cowboy never changes the rules itself: if you agree, edit cowboy.json.
• Scorecards: each agent, and people, by area of the code and by month.
• Needs a person: Cowboy's guesses, with yes and no buttons.

Careful pull requests also get more attention, so a pen's rate shows what happened, not what the pen caused.

Do the pens beat simple alternatives?

A pen that takes a quarter of your pull requests will catch some trouble by luck. So the outcomes page gives the same share of pull requests to simple alternatives and counts how many of the pull requests that had trouble each one caught:

• The careful pen against the largest pull requests first, the ones touching the most files first, and a random pick.
• The easy pen against the same share of the smallest pull requests.

Each result has a likely range (95%, from 2,000 resamples) and is said in words: outperformed (the whole range clears the line), no demonstrated advantage (it doesn't, which isn't the same as useless), or did worse. Until a repo has 20 pull requests with trouble, it says insufficient evidence and nothing else. It's the same calculation as the published evidence; here, every kind of trouble counts, not only reverts. Run cowboy evaluate for the same thing in a terminal, or cowboy evaluate --json for all the numbers.

Don't wait a month

On a fresh install the outcomes page offers rate the last 90 days (or run cowboy backfill --days 90). Cowboy sorts the pull requests that already merged, the way it would have, and checks what happened to them. Nothing is written to your git host: no labels, no comments. It uses today's rules and today's import graph, and these pull requests stay out of the tally.

Connect incidents

Each is optional. Point the tool at your Cowboy address:

{"id": "INC-12", "title": "Checkout errors", "severity": "sev2",
 "started_at": "2026-10-19T14:02:00Z", "url": "https://…",
 "repo": "acme/web", "release": "v2.3.0", "files": ["src/pay/api.ts"]}

Only title is required. If a tool names a project or service rather than the repo, map it with INCIDENT_REPOS=web-frontend=acme/web; with one repo, nothing is needed. For each incident Cowboy suggests up to three likely causes: shipped in that release (or just before it started), and changed files that appear in the error. It never uses the pen to pick a cause, so the pens are judged fairly. You can also link an incident to a pull request by hand.

The audit record

Every sort, move, merge, approval, outcome and confirmation is added to audit.jsonl next to the data file. Each entry carries the hash of the one before, so cowboy audit verify finds any entry that was changed or removed. The weekly tally post includes the latest fingerprint, which gives you a copy outside the server to check against. Export it with cowboy audit export --format csv, or from the outcomes page.

The benchmark

cowboy benchmark export writes counts only: pens, agents, rules and kinds of outcome, with no repo names, file paths, titles or people. Read it, and share it only if you want to. cowboy benchmark report a.json b.json pools several teams' files into one report. Cowboy never sends it anywhere.

How well it works

On two years of Sentry, Grafana and n8n history, Cowboy traced 98.8%, 87% and 70% of reverts to the pull request they undid; the rest name no pull request, title or commit. Replayed over 7,721 real merged pull requests from the same repos, the reverts it found matched an independent earlier count on 213 pull requests (8 found only there, 5 only here), and, weighted back to Sentry's real mix of pull requests, 1.42% of careful pull requests were reverted within 30 days against 0.81% of easy ones. Those repos are the ones the rules were tuned on; on Supabase, which the rules never saw, careful PRs were reverted about as often as any other, and on three fresh repos (next.js, Airflow, Kibana) neither revised rules nor rules fitted to each repo beat chance (the evidence page has all three rounds). Suggested fixes looked right in about 7 in 10 of a sample, which is why a person confirms them. Everything else has been tested against stand-ins for GitHub, Sentry and PagerDuty, not yet on a live repo for 30 days. See the check.

Habits and agent guide preview

The Habits page shows, for each repo, how often merged pull requests keep five habits, and whether each went with less trouble after merge in that repo:

• Tests with code: changes tests alongside source code.
• No shortcuts: no silenced warnings, skipped or deleted tests, loosened CI or catch-alls that hide errors.
• Reviewable size: under 400 lines of source.
• Reviewed before merge: approved by someone other than the author.
• Checks green: checks were passing when Cowboy last sorted it.

Each habit shows its share over the last 90 days and the 90 days before, and a verdict from all rated history: holds here, did worse here, no demonstrated difference or not enough evidence yet (fewer than 20 pull requests with trouble). Size is held fixed, and a verdict needs the whole 95% range to clear the line. When a habit shows no difference and the range is tight (0.8× to 1.25×), the page says a rule requiring it may be a ritual worth questioning; when the range is wide, it says it can't tell yet.

Numbers are for the whole team, never per person. Across nine public repos no habit went with fewer reverts everywhere, which is why each repo gets its own answer.

Guide your agents

Coding agents follow an instructions file in the repo (AGENTS.md, CLAUDE.md or .github/copilot-instructions.md). Cowboy writes a section for it: the paths and files that get a careful review, the size that gets the quickest review, the shortcuts it flags (only the checks you have on), this team's habits, and how to say an agent helped.

cowboy guide --repo acme/app                     # print it
cowboy guide --repo acme/app --write AGENTS.md   # put it in the file

The section sits between <!-- cowboy:guide start --> and <!-- cowboy:guide end -->, so running it again replaces only that part. It reads the repo's cowboy.json from the current folder (or --config). The Habits page has the same text with Copy and Download buttons. Cowboy never commits it for you.

What Cowboy writes and stores

On your git host

• Labels: cowboy:easy, cowboy:normal, cowboy:careful, cowboy:double, cowboy:stray, agent:<name> and agent:likely.
• One comment per agent pull request (and per careful pull request from a person), edited in place on every push, never a new one.
• Review requests to code owners, for careful pull requests.

That's all. It never merges, closes, approves or pushes code.

On your server

• cowboy.db next to DATA: each decision, its reasons and your team's moves, in SQLite (built in, nothing to install). Every change is saved in a transaction, so a crash never leaves it half-written. A cowboy.json from an older version is moved over on first start and kept as cowboy.json.migrated.
• baselines.json next to it: per-author history for the experimental agent check.
• tally-sent.json next to it, if you send the weekly tally: when each post last went out.
• settings.json next to it, if you connected on the setup page: the login (a token, or the GitHub CLI's), the repos, the mode and a webhook secret Cowboy made. Readable only by you.
• github-app.json next to it, if you connected with the GitHub App: the app's id, private key and webhook secret, readable only by Cowboy's user.
• outcomes.db next to it: merges, outcomes and incidents for After merge, also SQLite.
• audit.jsonl next to it: the tamper-evident audit record.

Over the network

Cowboy talks only to your git host (including downloading the default branch for blast radius) and, if you set them, your own model server (MODEL_URL) and where the weekly tally goes (your Slack webhook or mail server). The tally holds counts only, no code. No telemetry, no accounts, nothing sent to us.

Local model notes preview

An open-weight model on your own server can add notes to each pull request it sorts: whether the change stayed on task, a three-line brief, and where to look first. The model never picks the pen; the same pull request gets the same pen with or without it. Notes that mention a file the pull request didn't change are dropped.

It works with anything that speaks the OpenAI chat API: Ollama, vLLM, llama.cpp server, LM Studio or LocalAI. With Ollama on the same machine:

ollama pull qwen2.5-coder:7b
export MODEL_URL=http://localhost:11434 MODEL_NAME=qwen2.5-coder:7b
cowboy check-model     # is it answering?
cowboy serve

Labels go on first and the notes follow when the model finishes. Without MODEL_URL, everything else works with rules only. The notes have only been tried against a stand-in model so far, so check the wording with your own model before relying on it.

Command line

The dashboard also serves /healthz for health checks, outside the login, and /api/decisions as JSON for signed-in users.

Troubleshooting

Nothing shows up in the corral

• Check the log at start: it says "token works" or "can't list pull requests" for each repo in REPOS.
• Pull requests from people get a pen label but no comment unless they're careful, so they're easy to miss on GitHub; they do show in the corral. If you turned off sort_all, only agent pull requests are sorted.

Webhooks fail with 401 "bad signature"

The webhook secret doesn't match WEBHOOK_SECRET, or the content type isn't application/json. Fix it in the webhook settings and use GitHub's Redeliver button. Missed pull requests are caught by the sweep within 5 minutes either way.

Cowboy won't start: "the dashboard would be open to anyone"

Set UI_PASSWORD, or AUTH_PROXY_HEADER behind a sign-in proxy. See Settings.

"files depend on it" never appears

Open the rules page: it says how blast radius is working for each repo, and why not if it isn't (for example, an archive over the size limit). See Blast radius.

Rate limit messages in the log

Cowboy waits out short GitHub rate limits by itself. For longer ones it gives up on that call and the next catch-up sweep tries again.

macOS says the app can't be opened

The builds aren't signed yet. The install script handles this; if you downloaded the file yourself, run xattr -d com.apple.quarantine ./cowboy-darwin-arm64 once.

"This setup link has expired or was already used"

The link back from GitHub works once, within an hour. Go to /setup and press the button again. If GitHub already created an app, delete it in GitHub's settings first or give the new one a different name.

Setup says Cowboy is connected, but no repos show up

The app isn't installed on any repos yet. Open /setup and press pick repos.

Something else

Open an issue on GitHub with the log lines around the problem.

Limits of this preview

• One-click GitHub App setup is tested against a stand-in for GitHub's API, not yet against github.com.
• After-merge tracking has been tested against stand-ins for GitHub, Sentry and PagerDuty, and its revert detection against real history, but not yet on a live repo for 30 days. Releases are matched to pull requests by time, not by walking the git history.
• GitLab, Forgejo and Gitea support is covered by tests but hasn't run against a live server yet.
• Review requests to code owners and strays are covered by tests but haven't yet been seen on a live repo.
• Review requests go to individual people, not teams like @org/team.
• The weekly tally goes to Slack-style webhooks and email; Microsoft Teams and Discord formats aren't built yet.
• Data is kept in SQLite on one server. Running several Cowboy servers against one shared database isn't supported.
• The Habits page reads reviews and checks from live installs only; a backfilled history fills in the code habits first.
• Blast radius for Java and Kotlin reads import lines only (files in the same package use each other without one), and hasn't been checked against a reference tool yet.
• The check for unmarked agent pull requests is experimental and catches few hidden ones.
• The macOS builds aren't signed.

Cowboy is free and open source under the AGPL-3.0 license. Whatever's free today stays free.