Per-PR audit procedure
Shared by the audit and pr-queue skills. Each skill builds its own list of
PRs in §1 (audit: the PRs the user names; pr-queue: a searched,
ranked shortlist) and then works through that list here, one PR at a time:
fix it, push it, and ask the user to merge or close it. The user merges; you
never do. "The list" below means whichever list the skill built.
If the audit finds a problem shared across the list (e.g. an existing suite full of fakes), propose fixing that first in its own PR off main, then work through the list. When the user asks, sweep main into every listed PR before the one-by-one pass.
2. Check out and merge main
Start from a clean tree (git status). Never git stash — the stash stack is
shared with other worktrees and sessions.
gh pr view <n> --json headRepositoryOwner,headRepository,headRefName,baseRefName,maintainerCanModify,body,files,commits
gh pr checkout <n>
git fetch origin main && git merge origin/main
git submodule update --init -- submodules/distilled # checkout moved it to the PR's old pin After every checkout/merge, check git status for hook fallout: a
rewritten pnpm-lock.yaml (restore main's unless the PR changes deps), a
stray untracked distilled/ git worktree at the root (git worktree remove),
regenerated example files (e.g. the Prisma fixture's generated/). Never
commit any of it.
- Fork head repos can be named differently from upstream; read
headRepository/headRepositoryOwnerbefore pushing. - Resolve conflicts by intent. Main wins on code it has since refactored; port the PR's change onto the new shape. Many conflicts are formatting-only (main moved to oxfmt) — check whether main made a real change, keep the PR's logic, reformat.
- A clean textual merge can still be wrong. Read main's intervening changes to the files the PR rewrites and check them semantically (a dropped argument can silently disable behavior). Port the PR's tests into main's restructured test file and keep main's timeouts.
- A PR far behind main may no longer test anything — main may have fixed half the bug or changed the code path. Re-check after merging.
- Stacks: merge main into the base PR first and push, then merge the base
branch into the child. After the base squash-merges, retarget the child to
main; resolve its conflicts by confirming main's file equals the base's final version. - After each merge, other listed PRs touching the same files (
Providers.ts, barrelindex.ts) may conflict — re-merge main when you pick them up. - A fresh worktree needs
git submodule update --initandpnpm installbefore anything runs.
3. Review the code
Read the full diff against origin/main and apply AGENTS.md. The standards:
- There is a regression test. No test changes → add one, preferably by extending an existing test.
- Tests live in the existing suite for the resource (e.g.
Worker.test.ts), or extend an existing deployed fixture with a new route. A new one-off test file — especially one with mocks — fails the audit. Move pure-helper cases into the canonical unit file and delete the one-off. - Tests are end-to-end. Only
test.providerwithstack.deploy/stack.destroy, orbeforeAll(deploy(Stack))fixtures. No mocked HTTP/SDK/CLI responses, fake layers, direct lifecycle calls, or fake-cloud engine tests. Cause drift for real through the cloud API (e.g. a point-in-time restore swapped into place), then redeploy and verify. Use real local infrastructure where it helps (aregistry:2container, a kind cluster, Docker Postgres +cloudflared). Assume Docker is installed: neverskipIfon Docker availability; a missing daemon fails the test. A spy that wraps the real service is fine; a fake is not. - Also audit the existing tests the PR touches. If a suite already fakes what it claims to test, say so plainly — don't mention it in passing.
- Pure utilities are the exception: one shared implementation in
src/with a broad unit suite over in-memory fixtures is welcome. Where a real tool defines the behavior (git, docker), check the cases against it. - The test covers what a reviewer would worry about (e.g. a worker with the database bound, when the change is about replacing that database).
- Tests never modify checked-in files. Copy fixtures into the gitignored
packages/alchemy/.tmpand edit the copy (importalchemy/...there;@/doesn't resolve from.tmp). - Per-user isolation: no hard-coded account- or zone-unique physical
names. Use engine-generated names, or derive them from
stack.stage. - Reconciler doctrine, Typed Error Doctrine (tagged errors, never
Effect.fail(new Error(...))or catching catch-alls by status), noEffect.orDiein lifecycle ops, no raw Promise/node:fs. - Environment through
effect/Config. Inside Effect code, read env vars withConfig.String(...)(Config.withDefault/Config.optionwhen optional), neverprocess.env— wrappingprocess.envinEffect.syncis not enough.Effect.syncis for factsConfigcan't express (process.platform,process.cwd()). Module-scope test gates (skipIf(!!process.env.FAST)) are exempt. Fix violations in the PR's diff, including ones the contributor wrote.
Design review — answer these in the report before the user has to ask:
does a change replace a resource, and can that lose data (a database bound to
a worker)? Are names spelled out rather than abbreviated
(CertificateTransparencyAlerting, not CtAlerting)? What are the defaults?
Does it slow the happy path? Are values assumed resolved that could be
Outputs? Does it put things in the right namespace? Does it follow the
library's design rules, or is it really user error? Surface product decisions
(e.g. a change in how resources are named per stage) with their migration cost
rather than making them silently.
Fix what's fixable yourself — rewrite a contributor's mocked test into the right shape, and fix the problems you flag rather than only listing them. Keep the PR to its scope; out-of-scope gaps become follow-ups.
Distilled: before adding a patch, find the root cause upstream — the
vendor may have changed the schema on purpose. A new typed error goes in its
own distilled PR; once merged, point the submodule at the distilled main
commit, never a side branch. A distilled bump with breaking fallout gets its
own alchemy PR (fix(...) if it repairs something); trim the contributor's PR
back to its own files. Unexplained cross-provider diffs in a PR confuse the
user — remove them or explain them.
Bugs on main found while testing get their own PR, merged first.
Close or park:
- Recommend closing when superseded on main, a duplicate, or the wrong approach. Cite what supersedes it. For overlapping PRs, run the weaker PR's tests against the stronger PR's code, and credit the closed author.
- If a closed PR has one good unrelated part, suggest the author narrow it.
- If a bug can't be reproduced, ask the author for a repro, park it, move on. Treat upvotes as a reason to ask rather than close.
- Parked PRs wait for an external update; don't keep reporting "nothing changed" on them.
4. Update JSDoc
Every changed or new prop/attribute has field-level JSDoc (@default where
relevant), and the resource-level JSDoc has ### sections with
**Example:** snippets for the new behavior, metadata tags last. Run
pnpm docs:check-jsdoc, and pnpm docs:gen to check generation. The
generated pages under website/src/content/docs/providers/ are gitignored —
remove any a contributor committed.
5. Run the tests
- Run every touched test file in full, plus the suites covering the
changed source — not just the new case via
-t. Wide-impact changes get the broad suites (e.g. all oftest/Cloudflare/Workers, plustest:examples). Dev-mode changes also need the*.local.test.tssuites. - Prove it fails on main. Commit first, then put the
src/files back to main's version (git show origin/main:<path> > <path>), run the new case, confirm it fails for the right reason, and restore from git (never from a/tmpbackup — a crash loses it, andgit checkout <file>discards uncommitted edits). A test that passes on main isn't testing the bug: an identical redeploy plans a no-op and never reachesreconcile, an unchanged fixture reuses a cached image, dev mode skips the code path. Check the logs or timing to confirm the path actually ran.packages/frontend-frameworksis consumed fromdist/, so rebuild withnpx tsdownaround this check. - Run with
pnpm test <file> --profile testing. "0 files" or "0 tests" is an invocation bug (bad path, zsh word-splitting — use arrays orxargs), never a pass. - Credentials: for Cloudflare live,
set -a; source .env; set +a(pnpm download:envif missing). AWS live uses SSO; a ~100ms setup failure means an expired session — ask the user to runaws sso login. Never set or unsetCIfor tests: a profile-store migration can wipe~/.alchemy. - Flaky failures: re-run the file alone, run it on main, then alternate. Report
a verdict table:
Failure | Re-run alone | On main | Verdict. Lower--concurrency(e.g. 8) for a clean signal on big suites. A flake you fix gets its own PR. - Don't overlap two runs of the same stack: the second run's opening
stack.destroy()tears down the first's resources. - Runs against main and interrupted runs leak resources. Clean up (delete
out-of-band resources before
stack.destroy()) and verify out-of-band that nothing is left.OwnedBySomeoneElseon a named resource is a leftover from an earlier run — ask before deleting anything, offering a fullpnpm nukeor deleting only that stack/stage's resources. - After a crash: check for surviving test processes, files still swapped to main, leaked resources, and background agents' progress before rerunning.
- Long suites block the worktree. Advance the next PR in a separate worktree meanwhile. A large refactor can go to a background agent in its own worktree with "commit, don't push"; review its result before pushing.
- Before pushing, run the CI Check steps your change can break:
pnpm exec tsrs -b,pnpm lint,pnpm format:check,pnpm docs:check-jsdocandpnpm exports:check. They're in.github/workflows/check.yml. A new test file or fixture is type-checked too.
6. Push and update the PR
- Push every change as soon as it's made — the user reviews GitHub, not your local branch.
- Before each push:
git fetchand merge any new commits from the author; checkgit diff --stat origin/main...HEADfor unrelated files the repo-wide pre-commit formatter swept in, and revert them. Runpnpm exec oxfmt --checkon every file the PR changes and read the whole output. - Leave every PR with CI passing. After each push, wait for every check
on the new head to finish (
gh pr checks <n>). When one fails, read its log (gh run view <id> --log-failed), fix the cause, push, and wait again. Don't end your turn while a local suite is still running: wait on it witht3_background_wait, since ending the turn is your report. To wait on CI,watch_pull_requestthe PR after the push and end the turn; you're woken when its checks finish. Ask to merge only once they pass, thenunwatch_pull_request. - Make sure CI actually runs. First-time contributors' runs sit at
action_requiredand pushes don't start them. Read the whole diff first: approve the runs (gh run approve <id>) only when it doesn't touch.github/, scripts CI executes, or anything that reads secrets or sends data out. Otherwise ask the user. Don't approve runs on parked PRs. If no checks appear at all, find out why (a draft, a push that didn't triggerpull_request) and get them started. - "Ready" means: tests pass locally, every CI check on the final head has
passed, not a draft, mergeable. A pending, failing or empty check list is
not ready. Then
gh pr ready <n>.
Rewrite the description to show the developer experience — every PR you touch, even when the author's body looks fine. Explain the change through the code a user writes, not through prose or bullet points about the implementation. Contributor bodies usually arrive as Change/Tests/Checks bullet lists; replace them rather than appending to them.
- one plain sentence on what changes and why, no heading above it
- then
tssnippets of the user's code that is new or now works — the props, values and variants they can now write. For a fix or a changed API, a before/afterdiffof the user's code - check every API signature in a snippet against the source
- prose only for the "why" a snippet can't show, one or two sentences
- no
#/##headings, bullet dumps, Change/Tests/Checks sections or test plans - end with the attribution line your harness asks for, if any
Bad — describes the implementation:
## Changes
- Added `retention` prop to `LogGroup`
- Updated reconcile to call `putRetentionPolicy` when retention changes
- Added tests Good — shows the user's code:
`LogGroup` now takes a retention period, so logs stop growing forever.
```ts
const logs = yield* AWS.Logs.LogGroup("Logs", {
retention: "30 days", // or "forever"
});
```
Changing `retention` updates the group in place; it no longer replaces it. Write it to a file:
gh pr edit <n> --body-file /tmp/pr-<n>.md
# gh pr edit can fail on the deprecated Projects (classic) API — use REST:
gh api -X PATCH repos/alchemy-run/alchemy/pulls/<n> -F body=@/tmp/pr-<n>.md
gh api -X PATCH repos/alchemy-run/alchemy/pulls/<n> -f base=main # retarget Comments posted on GitHub must stand alone — no references to this conversation. State what's allowed, what was fixed, and why the rule exists, with a code example.
If the T3 link_pull_request tool is available, link every PR you touch or
open to this thread.
7. Report and ask
Keep it short — longer explanations only when asked. End the turn after each PR with:
### #N: <behavior>. I recommend merging.
<Why: the bug or gap, with the error text if there is one.>
<The fix, often a short diff.>
<What I changed on top of the author's work.>
| Test | Code | Result |
| --- | --- | --- |
| test/AWS/RDS/DBInstance.test.ts (full, live) | PR | ✅ 6/6 |
| … "refresh after restore" | main (expected to fail) | ❌ fails as expected: <error> |
Not run: <suites and why>.
<Design-review answers, decisions for you, follow-ups.>
Next in the list: #M (omit when it was the only one).
https://github.com/alchemy-run/alchemy/pull/N - Say which tests ran and whether they passed. Every CI check must have passed on the head you report before you call it "ready", but CI status doesn't replace the test report.
- Label deliberate runs against main as expected failures.
- Say plainly what was not run.
- When fixes span several PRs, say which PR holds each fix and give the merge order as a numbered list with one URL per line, closes included.
- When a decision is the user's, give numbered options with a recommendation.
- The last line of every message about a PR is its full URL — progress notes and questions included. Several PRs → all URLs at the end.
8. Keep the list moving
- Don't wait idle on a merge: start the next PR and keep the pending link in each message.
- On "merged, next": check it really merged (
gh pr view <n> --json state— the user sometimes names the wrong number), re-merge the new main into the next PR, then continue. - On "merge", "close", "park", "skip": update the table and act. Close only on explicit instruction, with a standalone comment crediting the author and naming what supersedes it.
- Answer "why" or "justify" questions directly with the error text and a diff snippet. "Refresh my memory" gets the status table plus open items.