Skip to main content

ADR 0001: AI review and evaluation act as merge gates

Status: Accepted (2026-08-04)

Context​

Every repository in the fleet runs an AI code review and an LLM-as-judge evaluation from the shared infra-ai-platform reusable workflows. Until now both were advisory: they posted comments and labels, and a PR could merge with an unaddressed blocker. The original inference provider (GitHub Models) was retired on 2026-07-30; the pipeline now runs on the Anthropic API (claude-sonnet-5 by default).

Decision​

  • A review that does not end in Recommendation: Approve fails its check (gate_on_recommendation: true). Comment counts as not-Approve by design.
  • The evaluation runs only after an approved review (it triggers off the review run's success) and fails its check when the judge scores the review below 9/10 (gate_min_score: 9). 9 is the floor of the rubric's Pass band (9–10 Pass, 6–8 Needs Improvement): anything the judge does not call Pass holds the merge.
  • Fail-closed on purpose: any failure of the review or the judge — including provider outages — leaves the check red. A red check for the wrong reason costs a re-run; a silent pass-through costs an unreviewed merge.
  • The manual re-run workflow stays ungated as a diagnostic, and carries its own check name so an ungated run can never satisfy a required check.
  • Nothing blocks until the checks are marked required in branch protection; the gates ship inert and are armed per repository.

Validation and rollback​

Validated end to end on the api-back canary: review Approve → recommendation gate green → evaluation 10/10. Harness defects found during the canary (a 3-line diff context that produced a false blocker, and a gate that parsed the previous round's verdict) were fixed in v2.9.1/v2.9.2 before fleet rollout, each with a regression test in infra-ai-platform.

Rollback is per repository and needs no revert: set gate_on_recommendation: false and remove gate_min_score in the three ai-*.yml callers, or unmark the checks as required. The evaluation always executes the default branch's copy of its workflow, so gate changes take effect only after they reach the default branch.