Mainfast Evidence
AI reviewers read the code. Mainfast Evidence checks the proof: did tests actually run on the commit being merged, did every required check pass, and did the agent touch what judges its own work (CI workflows, CODEOWNERS, test config, the tests)? It posts one evidence card per head commit.
What it catches
- Tests not run, failed, skipped or empty on the head commit. Evidence from an earlier push does not count.
- Required checks missing, skipped or neutral on the head commit (GitHub counts skipped as passing).
- Test weakening: deleted tests, removed test cases, dropped or commented-out assertions, added skip or focus markers, trivial assertions.
- Touched files that decide judgement: CI workflows, CODEOWNERS, test config, lockfiles, and the policy file itself.
- PR description claims such as "added tests" or "all tests pass" that the diff and checks do not back.
- Secret-like strings in added lines only (values are redacted), oversized diffs, and changed-line coverage against a threshold.
- Agent PRs are detected by author, commit trailers, emails, branch prefix, labels, PR body footers or git-ai notes. Other PRs pass as "Not an agent PR".
What you see
A real PR comment from a private test repository, where an agent branch removed a test and still claimed all tests pass.
Install
name: Mainfast Evidence
on:
pull_request:
types: [opened, synchronize, reopened, edited, labeled, unlabeled, ready_for_review]
workflow_run:
workflows: [CI]
types: [completed]
permissions: {}
jobs:
pr: # same-repository PRs, every push
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
permissions:
contents: read
checks: write
pull-requests: write
statuses: read
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
with:
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}
- uses: mainfast/evidence@v0
with:
wait-for-checks: "600"
ci: # after CI finishes; also the only job that sees fork PRs. No checkout.
if: github.event_name == 'workflow_run'
runs-on: ubuntu-latest
permissions:
contents: read
checks: write
pull-requests: write
statuses: read
actions: read
steps:
- uses: mainfast/evidence@v0
with:
local-diff: "false"evidence:
apply_to: agents # or all
test_checks: ["test", "/integration/"]
required_checks: [test]
coverage:
files: [coverage/lcov.info]
min_changed_lines: 80
flags:
touched_lockfiles: off
required_check_failed: failmainfast-evidence local --base main [--head HEAD] [--body-file pr.md] [--coverage lcov.info]
mainfast-evidence pr --repo owner/name --pr 42 [--post]
mainfast-evidence flagsStep by step, with what to check after each part: installation manual. Everything else: the README.
Permissions
- contents: read (both jobs): PR files and commits, the policy at the base commit, classic required checks and rulesets.
- checks: write (both jobs): create or update the check run on the head sha, and read other check runs.
- pull-requests: write (both jobs): the sticky comment, and finding a fork PR by head sha.
- statuses: read (both jobs): commit statuses count as checks. Without it a status-only test or required check reads as missing.
- actions: read (workflow_run job only): to download the coverage artifact from the CI run.
- No administration permission is needed. If the token cannot post, the job fails and the card goes to the job summary.
Privacy
- The Action runs no code from the PR in any job: it reads the API, parses text, and in the pr job runs git on checked-out objects, never a script, hook, build or test.
- The workflow_run job holds a write token, so the example has no checkout and no run: step; the diff comes from the API.
- The policy is read from the PR's base commit, never from the fork or branch, so a PR cannot loosen its own gate.
- Secret-like values are redacted in output. Nothing is sent to a third party; only GitHub API calls with your token. With a licence key, only the key and the organisation id go to Polar to check it.
- Coverage artifacts from fork PRs are ignored, and git-ai notes are never read for fork PRs.
- Paths, check names, description quotes, branch names and commit messages are escaped and length-capped.
Limits
- Agent detection can miss an agent that leaves no signal; the README lists the signals. A missed detection treats the PR like any other, and no signal can loosen the check.
- A first-time contributor's fork CI may wait for approval; until it runs, the fork PR has no card.
- Coverage is not available on fork PRs: the report comes from the fork's own CI.
- Test-weakening rules cover JavaScript/TypeScript, Python and Go only.
- Without a full checkout the diff comes from API patches, which GitHub truncates for large files (diff_unanalyzed).
- extends: is accepted but not applied yet.
More in Mainfast
Install Mainfast Evidence
Proof, not review, for pull requests written by coding agents.
