Jimmy Van Veen

Auto-Merge for a Fleet of One

I copied Amplitude's risk-scored auto-merge onto my own repos. Two weeks later it had merged one pull request by itself, and the reasons were more useful than the feature.

Amplitude published a post about tripling pull request throughput in six months without hiring anyone. The headline is the 3x. The number I could not stop thinking about was cycle time, which fell from 5.2 hours to 44 minutes.

They got there in two moves. First they made CI fast. Frontend runs went from about 30 minutes to 3 or 4. Then they stopped putting a human in front of changes that did not need one. Every PR gets scored on risk, using size, scope, where in the codebase it lands, test coverage, and whether it touches a public API or a data model. Low risk merges on green. Medium and high route to a code owner.

I do not have code owners. I have me. I am the author, the reviewer, the approver, and the person who has to do this after a day job.

The bottleneck is me

Since I let coding agents write most of my side-project code, writing the code is no longer the slow part. The slow part is that every change queues behind one person, and that person also wrote it and already knows what is in it.

So the review I perform is mostly theatre. I open a PR I dictated an hour ago, skim a diff I already read, and click merge. Except sometimes I do not click merge for three days, because it is Tuesday and I am tired, and by Friday there are six of them stacked up and I have lost the thread on all of them.

Amplitude's answer does not depend on having a big org. Decide in advance which changes deserve a human, then let the rest go.

Three tiers

I collapsed their model to one question. Which changes need my eyes before they land, and which should never wait on me?

Low is every changed file matching a low-risk glob, meaning docs, tests, formatting and editor config, with a diff under 300 lines. It merges on green.

Medium is ordinary source code. It merges on green plus a clean verdict from the review bot. The bot posts one comment and its last line has to be exactly VERDICT: clean. Anything else and the PR sits.

High is anything under .github/, anything matching a per-repo protected glob, 600 or more changed lines, or more than three top-level directories touched. A human, always. The .github/ rule is not configurable by the caller, because a PR must not be able to argue itself down a tier by editing the classifier.

That rule fired on day one. The system labelled every PR that rolled it out to a repo risk:high, including the one adding it.

The workflows live in one public repo, JimmayVV/fleet-ci. A project calls them in a few lines.

1jobs:
2  ci:
3    uses: JimmayVV/fleet-ci/.github/workflows/node-ci.yml@v1
4    with:
5      package-manager: bun
6      playwright: true

Change a rule in one place, move the v1 tag, and every repo picks it up on its next run. New repos start from JimmayVV/fleet-template.

What broke on day one

Dependabot merged two PRs in twenty seconds

My poker clock repo is private on a personal account, and GitHub only allows rulesets on private repos on a paid plan. I wired up Dependabot auto-merge anyway and watched two dependency bumps open and close before CI had finished installing. PR #50 opened at 19:20:08 and merged at 19:20:28.

gh pr merge --auto does not mean "merge on green". It means "merge when nothing is blocking", and on a branch with no required checks nothing is ever blocking. Auto-merge becomes merge-now, with no error.

The fix is a guard every arming step runs first. It asks the API whether the branch has any rule requiring status checks, and refuses to arm if not. I would rather auto-merge not work at all than work by accident.

Dependabot does not write bun lockfiles

Every Dependabot PR on the bun repo failed with lockfile had changes, but lockfile is frozen. Dependabot edits package.json and leaves bun.lock alone. The two PRs that merged in twenty seconds carried that mismatch into main.

You cannot read secrets in a reusable workflow's with:

build-env: VITE_CONVEX_URL=${{ secrets.VITE_CONVEX_URL }} failed to parse. The secrets context is not available there, so the run showed up with a file path instead of a name. The fix was a real secret input on the reusable workflow.

The review bot will not review the PR that adds it

The Claude review job on the rollout PR went green in twelve seconds and did nothing. GitHub requires the workflow file to already exist on the default branch. That is a sensible guard, but a green check with no verdict behind it looks exactly like a passing review.

Two weeks later

I promised myself numbers, so here they are. Eight repos, September 14 to 28, 80 pull requests.

  • One PR merged the way I designed it to. A patch bump to image-size on this site got labelled low and merged 3.5 minutes after it opened, with no human involved.
  • Two more merged by bot. Those were the twenty-second accidents.
  • Twenty I merged by hand. That includes every medium-risk PR. Not one cleared the review bar on its own.
  • Forty-four were still open. Forty-one of those were from Dependabot.

The design was "decide in advance which changes need a human, then let the rest go." Almost nothing got let go, for three reasons.

Most of the fleet cannot auto-merge at all. Five of the eight repos are private on free plans. No rulesets, so the guard correctly refuses every time. Those repos get labels and reviews, and I merge everything.

The guard was right by accident. On those private repos the check for protection returns a 403. The line meant to turn a failure into zero, gh api ... 2>/dev/null || echo 0, kept GitHub's error text on stdout next to the zero. The comparison after it errored, and the script fell through to "not protected", which happened to be the correct answer. The one piece of this I would have called clever was working for the wrong reason.

Nineteen Dependabot PRs had no risk label. I had put the labelling inside the step that arms auto-merge, and that step only runs when the guard passes. On a private repo, patch bumps got no tier at all. Grouped and security updates, where Dependabot does not report an update type, fell through every branch. So the queue I was supposed to triage could not tell me which PRs mattered.

CI was the one clear win. The median run was 3.2 minutes on this site and 2.1 on the poker clock, both under the five-minute bar I set.

The fix, and the part that does not need one

The fix is in fleet-ci#10. Labelling is its own step now, and it runs before the guard. Patch and minor bumps are low. Everything else is high, with a warning that says why. Arming follows the label. The guard only accepts a bare integer.

That does not make private repos auto-merge. Nothing free does. It sorts their queue, so clearing it by hand takes minutes instead of an evening.

The twenty-two high-risk Dependabot PRs are the part that needs no fix. They are major version bumps, actions/checkout from 4 to 7 and the like. The system put them in front of me because a human should read them. That is the tiers working.

What the numbers decided

Counting the fleet showed me it was organised by accident. Products sat next to experiments on my personal account, all private, all unable to use the thing I had built for them.

So I moved them. Personal open source stays on my account. Products go to Redline Labs, a GitHub organisation that will pay GitHub's roughly $4 a month for private-repo rulesets when one of those products earns it. The automation is free. Turning it on for private code costs four dollars. I only learned that was the real trade by counting.

Why it is free

I did think about selling it. Mergify is free for up to five users on private repos. Aviator is free under fifteen developers. CodeRabbit's free tier has no repo or seat cap. Graphite's Hobby tier is aimed at individuals reviewing agent-written code.

Four funded companies looked at solo developers and decided we are an acquisition channel, not a revenue line. There is no price under zero. And fleet-ci is YAML, some globs, a line count, and a regex over one line of a comment. A competent engineer rebuilds it in a weekend.

So it is public, it is free, and if it saves you finding these bugs yourself, that was the point. I will report again once the fix has had a month to run.

github.com/JimmayVV/fleet-ci and github.com/JimmayVV/fleet-template.