Code review bottlenecks: How we hill climbed our way to higher PR throughput

After we adopted coding agents at Assembled, engineers kept telling me they felt much faster. But when we looked at our delivery metrics, the improvement was much smaller than we expected. We were writing more code and creating more PRs, but only merging slightly more of them. Time to first review had increased from a median of 3.5h to over 16h, and it was becoming normal to see an engineer with dozens of PRs waiting in a single Graphite stack.

We dug into our engineering productivity data and found that human code review had become the bottleneck. So we built an automated reviewer that approves low-risk PRs. The early results are very promising and showed PRs merged reaching 2.4× of our pre-agent baseline. Most surprisingly, throughput for large PRs increased 3.5×, even though the auto-reviewer generally did not approve them. Much of the new throughput is centered on bugfixes (3.7x above baseline) and refactors (7.5x above baseline), but we also saw accelerated feature development (2.5x above baseline).

This post covers how we found the bottleneck, how we built the auto-reviewer, and our hypotheses for why it worked as well as it did.

Reviewers couldn't keep up with agent-generated code

For some context, we sell customer support software to companies like Salesforce and DoorDash, as well as organizations like the State of Georgia. Hundreds of thousands of support agents rely on our product being stable. Vibe coding without verifying correctness was never an option for us. Still, we felt like we should be moving faster given the coding agents and other AI tools we now had access to.

So we did an investigation into our engineering productivity. As we dug into the metrics, we found a few things:

• The number of open PRs was much larger than before, and the majority of them weren't getting merged (and if they were, they weren’t getting merged quickly).
• Stacks were getting deeper. Engineers were breaking big features into reviewable pieces, which is the right instinct, but it regularly produced stacks of 20 PRs for a single project.
• Time to first review was up from a median of 3.5h in December 2025 to over 16h in May 2026, driven by a much larger incoming PR count.
• PR size had increased 2.5x, since agent-written diffs tend to be larger than the human-written version of the same change.

There were also costs that didn't show up on a dashboard. Engineers had to multiplex across several agent jobs to stay effective, which meant people who were used to concentrating on one thing were now juggling several. More non-engineers were writing PRs (which we wanted), but those PRs didn't always follow established engineering conventions, so reviewing them carried emotional burden.

Every property of coding agents points toward more code, but we were still routing all of it through the same number of human reviewers.

Setting up an automated reviewer for low-risk PRs

To alleviate this, we started with the safest possible experiment: auto-approving test-only PRs. It worked fine, but there were relatively few test-only PRs (only a handful per week), so it didn't move the needle perceptibly.

Then we got more aggressive and built a review system directly into 143.dev, our internal coding agent platform [0]. The system:

• Runs two independent reviews. Codex (GPT-5.6-sol, high) and Claude Code (Claude Opus 5, high) each review the PR in separate cloud sandboxes. We intentionally made the prompts simple, using the built-in /review command. OpenAI and Anthropic have enormous resources making sure their agents work well for review, so we leveraged that as much as possible.
• Makes an approval decision. An orchestrator agent (GPT-5.6-sol, high) pulls in both reviews, applies our approval policy, and decides whether the PR gets auto-approved. The approval policy includes both deterministic rules (e.g. only approve PRs under 1000 lines changed) as well as LLM-based policies (e.g. don’t approve any changes in credential or secret handling or cryptographic key management).

We spent a large amount of time tuning the policy that we use, landing on something similar to the following:

• A good PR description
• High-quality testing evidence: screenshots or videos for product changes, or direct evidence that the change does what it claims
• An implementation that follows established repository patterns
• No changes to risky areas like authentication, billing, or permissions
• No new architectural patterns
• A few other guardrails based on failure modes we saw during rollout

Iterating on the policy

As Greg Mankiw says: “People respond to incentives” and the auto-approval policy was the ultimate version of this. We saw that our engineers ended up shaping PRs for the approval policy, so whatever the policy rewards, you get more of.

This meant we had to be very thoughtful about what we wanted to let through. We realized that the only usable version of an auto-approval policy was one with high integrity and broad coverage. When there was low coverage (e.g. it doesn’t auto-approve enough things), the reviewer didn’t get enough traction to be used and people abandoned or forgot about it.

At the same time, we needed to ensure we weren’t approving low quality PRs either:

• Policy too loose: people route work through it precisely because it doesn't look closely, mistakes compound in master, and we have an incredibly dangerous buildup of bugs.
• Policy too strict: it fails silently and you never recover usage. Engineers try it twice, get rejected, and stop asking, which means you save no review time and stop learning where the policy is wrong.

So the job was to hold a narrow ridge between the two cliffs.

Rolling out safely

We found the best way to make adjustments to the policy to tune it towards the “Useful automation” region was a phased approach:

• Start with trusted alpha testers and review everything. We launched with a small group that reported both false positives and false negatives, with much more scrutiny on false positives. Before a wider launch, a mistaken approval would still be noticed by the group. The team that owned the policy also reviewed every PR as they came in and compared its decision with the system’s. Surprisingly, we found that our initial policy was far too conservative. We had added a lot of restrictions that were not buying us meaningful safety, and the reviewer rejected so much that engineers were unlikely to keep using it.
• Expand gradually, with audits and direct feedback. Once we were confident in the initial behavior, we rolled it out more broadly.
  • We started with daily reviews of approval quality, then moved to a regular cadence as usage and confidence grew. We also built automations that scanned review sessions and identified recurring failure patterns.
  • Engineers could request a re-review or leave free-form feedback directly on the PR. The 143 reviewer categorized that feedback for the admins, who could then decide whether the policy needed to change.
• Track usage and approval rates by engineer. We added a dedicated page in 143 showing who was requesting the most auto-reviews and what percentage of their PRs were being approved.
  • This was unusually useful because aggregate approval rates hid very different behaviors. We could talk to power users to understand how they were using the system (and whether they were finding ways to game it). We could also talk to infrequent users to understand what was keeping them away.

The goal was to begin with a narrow set of changes where we could establish trust, then gradually expand coverage as our evidence and guardrails improved. The policy needed to be strict enough that an approval meant something, but useful enough that engineers would actually change their behavior to qualify for it.

Results: 2.4× more PRs merged

The metrics from our rollout were incredible and they far exceeded our initial expectations:

• PRs merged rose to 2.4× of our pre-agent baseline. Lines of code rose by 2.8×, so we weren't just slicing the same work into more PRs.
• Auto-approvals grew to 43% of merges over the three weeks after the full-team launch.
• Large PRs (600+ lines) increased 3.5× in throughput, with p90 wait time holding steady.
• Refactoring went up 7.5× and test work went up 3.4×. Nearly half of our peak week was debt paydown we'd previously never gotten to.
• Bugs reported per week went down 27% and reverts per PR stayed steady at 0.8%. Note this window is too short to be conclusive, especially because of the time delay of bugs being reported, but it is encouraging and these are numbers we’ll continue to check.
• Non-engineers accounted for 13% of all PRs. We had a stricter approval policy set up for designers, PM, and customer support, but we still allowed auto approval for small changes (updating copy, fixing bugs, etc.). The auto-reviewer, combined with cloud agents like Devin, allowed non-engineers to contribute significantly without requiring a large amount of engineering handholding.

These are observational results from a rollout, not a randomized trial, and we don't think the auto-reviewer gets credit for every point of movement (and there are still observations of bugs and long-term quality that need to be made). But the timing, the composition of the extra work, and what engineers told us point to a few mechanisms at work:

Fast, predictable review changed which work people attempted. The median auto-review takes eight minutes, and just as importantly, you know roughly when it's coming and what bar you have to clear. Before, an engineer who noticed a minor refactor or a missing test had to decide whether it was worth consuming a human reviewer's attention. These PRs were easy to deprioritize because they weren’t urgent. With a reliable fast lane, engineers became much more willing to spin off small improvements. If you're in an area of the codebase and see something small worth fixing, it's now easy to kick off a separate job and get it reviewed quickly. This is a big part of why refactoring grew 7.5x while overall output grew 2.4x.

Much of the new speed centered on bug-fixes and performance work. Instead of unleashing a bevy of bad PRs, the auto-reviewer has actually encouraged better code because of the ease at which smaller bugfixes and targeted performance work move through the system. We’ve seen our primary database CPU usage decrease by 20% as teams set up automations to improve slow queries they owned.

The policy improved the PRs themselves. To qualify for auto-approval, a PR needs a real description, direct testing evidence, and an implementation that follows existing patterns. Engineers adapted to this quickly. More PRs showed up with testing evidence at the top and screenshots of the change, and large changes got broken into smaller units so each piece was more likely to qualify. This raised the baseline quality of all PRs, including the ones humans review, and it pushed the codebase toward smaller, cleaner changes, which is something we wanted anyway.

Removing small PRs from the queue helped the large ones. This was the most counterintuitive result, since large PRs sat outside the fast lane of auto-approvals. Our best explanation is that review capacity is about concentration as much as hours. Reviewers were no longer being bombarded by small, conventional PRs all day, so when a PR did land in their queue, they knew it actually needed their attention and could spend real time on it. That freed-up attention showed up as 3.5× more throughput on the PRs that still required human judgment.

Costs

Across a 14-day sample, our automated code reviews consumed an estimated $3.57 in LLM usage per attempt. That’s more than a lightweight review bot but substantially less than premium managed multi-agent review (e.g. Anthropic’s managed code review), while buying two independent frontier-model assessments and a separate synthesis pass.

Our internal system is also connected to OpenCode and we can relatively straightforwardly swap out any of the reviewer models with other cheaper model types in the future (e.g. Kimi K3 or Qwen3.8). The framework we use allows us to have any number of coding agent reviewers under the orchestrator so this is tunable in the future.

What's next

The results are early, so we still need to understand how these auto-reviews affect our long term outcomes like bugs shipped, product quality, roadmap speed, etc.

We're continuing to tune the auto-reviewer: categorizing the reviews coming in, identifying common failure patterns, and feeding our recurring failure patterns back into the policy automatically.

Speeding up one stage of a pipeline tends to move the constraint to the next stage. As agents make implementation and review faster, we expect the constraint to move again, maybe to specification, testing, or deciding what's worth building. When it does, we'll continue hill climbing there.

If you're interested in helping us build self-improving coding agents, we're hiring!

‍

Thank you to Alec Rodgers, Chasen Stark, and Paul Hager for reading drafts of this post.

‍

[0] Why didn't we use a commercially available reviewer? There were 3 main reasons a) our own system has really good price to performance characteristics because we can choose exactly what models we want to run, especially frontier models without paying extra premiums b) we also liked the control: our team has found that Codex and Claude Code together catch bugs better than a single reviewer and c) owning the system lets us adjust our approval policies automatically over time.

‍