The AI Wrote the Code. Was It Any Good?
An AI agent generates 1,000 lines of code in minutes. CI fails repeatedly, the change adds a library that duplicates existing functionality, and the reviewer rewrites most of the implementation. Should the team count that as a productivity gain?
Lines generated and completion counts measure activity: how much the tool was used. They do not tell us whether the change was safe or reduced work across the team. DORA’s 2025 research similarly describes AI as an amplifier of organizational strengths and weaknesses; without robust testing and fast feedback loops, greater change volume can expose downstream instability. DORA’s 2025 research
This article uses the following definition:
AI-generated code is high quality when its verification results are reproducible, it passes the team’s existing standards, required human intervention remains acceptable, and it causes little post-merge rework.
The primary measurement target is therefore not output volume, but the cost of accepting a change safely.
1. Why Generated Lines Are Not a Quality Metric
Generated code may be deleted, regenerated, or manually edited before adoption. Redundant abstractions and duplicate tests can increase line counts without adding value. Larger diffs may instead expand the review surface and create more places for defects to hide.
Making AI usage rate a target also creates perverse incentives. Developers may invoke AI for changes that need only a small manual edit or generate code that the product does not need.
Generation volume can still provide context. It should not be the KPI. Thirty minutes spent reviewing 50 lines of configuration is not equivalent to 30 minutes spent on 50 lines of authentication logic. Interpret diff size together with change type, affected module, and risk class.
The relevant question is not how much the AI wrote. It is how cheaply and safely the team could accept the resulting change.
2. Evaluate Change Sets, Not Isolated Generated Code
Precisely tracking which lines came from AI and where human editing began is difficult. Use the pull request as the default evaluation unit instead. Commits, development tasks, individual agent runs, and production releases are possible alternatives, but a PR naturally connects the diff, CI evidence, review activity, and post-merge history.
For each PR, retain at least:
- Whether and how AI was used
- Change type, affected modules, and diff size
- A risk class for areas such as authentication, payments, or data migration
- AI rerun count and the author’s manual-edit category
- CI results, including the first run
- Direct and transitive dependency changes
- Active review time and revision rounds
- Related reverts, hotfixes, and follow-up changes
Do not compare all AI-assisted and non-AI PRs as two undifferentiated groups. If AI is reserved for difficult work, that comparison penalizes the AI group. Compare within strata that align change type, size, risk, system, developer experience, and greenfield versus maintenance work.
3. Quality Gate 1 — Tests: Did They Pass, and Can We Trust Them?
Passing tests are necessary, but not sufficient. When AI misunderstands a requirement, it may encode the same misunderstanding in both implementation and tests. Coverage percentages and generated-test counts alone will not reveal that failure mode.
Hard gates
The following generally make suitable merge requirements:
- Build successfully in a clean environment without relying on caches
- Pass existing tests, static analysis, and type checks
- Add or update tests for changed behavior
- Cover risk-relevant boundaries, error paths, permissions, and concurrency
- Produce stable results when the same commit is run again
Reproducibility requires treating build tools, configuration, dependencies, time zones, and locales as inputs—not just source code. Reproducible Builds documentation likewise identifies environment definition and sources of variance such as timestamps, ordering, randomness, and build paths. Reproducible Builds documentation
Observed metrics
First-run CI success rate is the percentage of PRs whose first CI execution succeeds. Unlike final success, it exposes how complete the initial change set was.
CI rerun rate is the percentage of PRs rerun without a code change. If the same commit alternates between success and failure, investigate the test infrastructure before blaming generated code. Google’s testing guidance notes that tests producing both outcomes against the same code become untrustworthy. Google Testing Blog
Test-gate false-positive rate measures stopped changes that investigation later classifies as having no code defect.
False-positive rate = incorrectly blocked changes ÷ all changes blocked by the gate
Post-merge defect escape rate is the percentage of accepted changes linked to a defect within the chosen observation window. If CI success remains high while escapes increase, improve the tests’ ability to represent requirements rather than merely adding tests. For critical logic, mutation testing can help determine whether a suite detects deliberate behavioral changes.
4. Quality Gate 2 — Dependencies: Working Code Is Not Necessarily Maintainable
AI may solve a local problem by adding a package instead of finding an existing utility. Each dependency also adds future costs involving vulnerabilities, licenses, upgrades, build size, and eventual replacement.
Review dependency changes for:
- New direct dependencies and their transitive additions
- Why existing functionality or the standard library is insufficient
- Maintenance status and update policy
- Known vulnerabilities, licensing, and distribution constraints
- Unintended major upgrades or broad lockfile churn
- Pinned versions and a reproducible retrieval path
Use hard gates for known vulnerabilities at or above the team’s rejection threshold, prohibited licenses, packages of unknown origin, and non-reproducible version declarations. The justification for a new dependency or a large lockfile change usually works better as a soft gate. GitHub’s official documentation provides an example of reviewing direct and indirect dependency changes in a PR and failing checks based on vulnerability severity or license. GitHub Dependency Review
Track new dependencies per PR, review findings about additions, security-driven rejections, and the share of dependencies removed or replaced soon after introduction. That last metric can signal unnecessary complexity, but should not determine failure by itself.
5. Quality Gate 3 — Human Intervention: Measure What People Had to Fix
AI’s value becomes visible in where and how much human work remains. Separate author intervention from reviewer intervention.
Author intervention
- Correction requests and AI reruns
- Manual editing after generation
- Generated implementations discarded and rewritten
- Switching from AI output to human implementation
Reviewer intervention
- Revision rounds and review comments
- Active time spent reviewing
- Requests for design changes
- Implementation rewritten by the reviewer
The most useful time measure is active review time, not elapsed time from PR creation to merge. Elapsed time includes queues, nights, and interruptions. Teams can begin with explicit start and stop actions, active time in the IDE or review UI, or a short self-report. Aggregate this for process improvement; using it for individual performance evaluation will distort the data.
When line-level rewrite attribution is impractical, use an ordinal scale:
- Accepted almost unchanged
- Minor edits
- Partially reimplemented
- Mostly reimplemented
- Discarded entirely
High intervention is not automatically failure. Human involvement is expected in authentication design or data migrations. Compare work within the same risk class and combine implementation, active review, and correction time.
6. Four Outcome Metrics That Show Whether the Gates Work
Tests, dependency checks, and human intervention are leading quality controls. Validate their effect using four outcome metrics.
6.1 Reproducibility
Measure the percentage of changes that produce the same build and test results from the same code and declared configuration in another clean environment. Include dependency restoration, repeated execution, execution order, time zones, external APIs, and implicit local-file dependencies.
6.2 False-positive rate
Measure the share of blocked changes that were not defective. Separate real defects stopped from valid changes blocked incorrectly. A noisy hard gate encourages warnings to be ignored or bypassed, so be prepared to return it to soft-gate status.
6.3 Rework rate
Measure changes requiring a revert, hotfix, same-cause follow-up PR, missing-requirement implementation, dependency removal, or performance or security repair within a defined post-merge window. DORA likewise treats deployments requiring immediate intervention and unplanned deployments caused by production defects as stability measures. DORA software delivery metrics
6.4 Review time
If an author saves 30 minutes but review and corrections grow by more than that, team-level acceptance cost has not improved. Track the full set:
Acceptance time = implementation time + active review time + correction time + post-merge rework time
A financial model may add role-specific labor rates and CI cost. Keep the component measures visible instead of relying on one score, so the team can see where work moved.
7. Rolling Quality Gates Out to a Team
Turning every signal into a merge requirement at once creates false positives and operational friction. Progress through observation, soft gates, and hard gates.
Phase 1: Observe
Use two to four weeks as an initial trial and collect first-run CI success, new dependencies, active review time, revision rounds, rework, and AI usage. Do not block merges yet. Establish baselines by repository and risk class.
Phase 2: Introduce soft gates
Warn about new dependencies, missing tests for critical logic, oversized diffs, and missing required reviews. Require a reason at merge time. Exception reasons become data for improving the rule.
Phase 3: Promote only stable controls to hard gates
Require items with clear decisions and low false-positive rates: successful builds and tests, no prohibited vulnerabilities or licenses, clean-environment reproducibility, and required reviewer approval.
Keep review time, comment count, AI reruns, rewrite category, and rework rate as observational metrics. Uniform thresholds invite gaming, such as avoiding difficult changes or withholding useful comments.
Review prevented defects, false positives, exceptions, and measurement overhead on a regular cadence. NIST’s SSDF similarly presents its practices as a risk-based foundation for incremental improvement—not a fixed checklist independent of organizational constraints. NIST Secure Software Development Framework
Conclusion: Move from Output Volume to Acceptance Cost
Generated lines and usage counts describe adoption, not quality. Ask whether results are reproducible, dependencies are justified and safe, human correction and review remain acceptable, and post-merge rework is not increasing.
The quality of AI-generated code is determined not by how much the AI wrote, but by how safely and cheaply the team could accept the resulting change.