dev-deploy-safely

Files (5)

• SKILL.md
• gate-merge.sh · scripts/
• pr-checks-green.sh · scripts/
• test_gate_merge.sh · scripts/tests/
• test_pr_checks_green.sh · scripts/tests/

SKILL.md (first lines)

# Dev Deploy Safely

Use this skill before production, shared staging, or preview deploys.

The core production rule: the branch doing a production deploy must contain every change that is already live or already production-bound.

For shared staging, decide first whether the environment is a production mirror,
QA/demo target, or release-candidate environment. Production mirrors and release
candidates should follow production gates. QA/demo staging can intentionally
diverge, but the deploy record must say what branch/commit is deployed and why
drift is acceptable.

For ephemeral previews, the goal is traceability rather than production parity:
record the branch, commit or artifact, URL, checks run, skipped checks, owner,
and cleanup path.

Fetching tells you what exists remotely. It does not prove your local deploy tree is safe. Deployment tools often publish the files or image from the current worktree, so stale local state can overwrite live fixes from another branch.

## Non-Negotiable Gates

1. Classify the target environment before choosing gates: production, shared staging, or ephemeral preview.
2. Fetch and prune before production or shared staging deploy planning.
3. For production, prove the deploy branch includes the latest remote default branch.
4. For production, check whether local default branch has unpushed production-parity commits.
5. For shared staging, branch freshness and local-default parity are mandatory only when staging mirrors production or is being used as a release candidate. Otherwise record intentional drift.
6. For ephemeral preview, record the branch/commit/artifact and target URL. Default-branch parity is not required unless the preview is used as a release candidate.
7. Audit active worktrees before multi-file or high-risk production/shared-staging deploys.
8. Commit and push the exact tree that passed tests before production deploy. For shared staging, strongly prefer a pushed commit. For preview, at minimum record the local commit or artifact digest.
9. Do not deploy production from a dirty worktree. Staging/preview dirty deploys are allowed only when intentionally testing local changes and the record says so.
10. If another branch has deployed first to the same production or shared staging surface, merge that branch or the live environment diff into the deploy branch, retest, then deploy.
11. Write a deploy manifest for file-based deploys.
12. Pull and diff live files before overwriting production or shared staging files.
13. Back up live files or retain an artifact rollback pointer before replacing production or shared staging.
14. Verify the target environment with smoke checks immediately after deploy.
15. Record what was deployed, how it was verified, which gates were skipped, and how to roll back or clean up.
16. On platforms that split traffic across revisions (Cloud Run, App Engine, Lambda aliases), a production deploy is NOT complete until the new revision serves **100% of live traffic** and you have smoked the **live production URL**. A revision deployed with no traffic and never flipped is **deployed dark**, not a finished deploy - see "Definition of done" below.
17. Before merging any change you classify as non-deploy (docs, skills, config), prove the merge does not deploy. See "Does merging this deploy?" below. If it does deploy, it IS a production deploy: run these gates or get explicit approval.

## Definition of done: live means 100% traffic, not just a new revision

"Deploy to production" and "put it straight to live" mean one thing: **the new revision serves 100% of production traffic on the real production URL, and you verified it there.** They do NOT mean "a new revision exists in production."

On traffic-split platforms the safe recipe deploys the new revision with **no traffic** first (a dormant revision), verifies it, then **flips traffic to it**. That flip is the step that makes the change live. Skip it - or stop after a green check on the revision's own tag/preview URL - and the new revision sits at **0% traffic** while the live URL keeps serving the old code. That is "deployed dark": present in production, serving nobody.

- Leaving a revision at 0% traffic is valid ONLY when a staged, canary, or dark rollout was explicitly requested. Then say so, record it, and hand over the single traffic flip that is still owed.
- Absent that request, flipping to 100% and smoking the live URL is mandatory before you call the deploy done.

> **Word of caution on "dark".** Teams often also use "dark" for a feature shipped behind an OFF flag (code live at 100% traffic, feature switched off). That is a different thing from a 0%-traffic revision. Both are "not actually live"; say which one you mean.

**The deploy's own record is part of "done", in the same run.** A doc that only records or reflects the deploy you just shipped and verified (writing the deploy record, flipping a deploy-log entry from PENDING to LIVE, an OWED-to-DONE QA line) is completed in the SAME run as the deploy, not left as a follow-up. It never covers new code or any other change.

## A permission wall is not a stop signal

On a headless or shared machine the ambient cloud identity is often narrow on purpose, so a gate read (build history, revision describe, image digest, logs) can fail with 403 / `PERMISSION_DENIED`, or return an empty list. Before you mark a gate as blocked, hand off, or ask the owner to log in with a personal account:

1. Check whether your environment provides an elevated, read-capable impersonation identity (your machine runbook or environment names it). Use only the identity it names. Never look up or guess other service accounts you might be able to impersonate. If there is none, the normal hand-off applies.
2. If there is one, re-run the read as that identity: `--impersonate-service-account=<sa>` on gcloud, or the `CLOUDSDK_AUTH_IMPERSONATE_SERVICE_ACCOUNT=<sa>` prefix, which also covers `gcloud storage` and bq. Python Google clients ignore both: build impersonated credentials in-process (`google.auth.impersonated_credentials`). Confirm the re-run printed the impersonation warning (or otherwise ran as the elevated identity); if it did not, the tool ignored the setting.
3. An empty result from the narrow identity is not proof that nothing exists. Re-run it impersonated before you trust it.
4. Impersonation gives capability, not approval. Every gate in this skill still applies, and so does your confirm-first list (authz, secrets, data, migrations, anything irreversible, external comms). Never log in with a personal account on a headless box: it cannot reauthenticate unattended, and it can poison a shared config.

## Does merging this deploy?

A merge to a default branch can deploy production even when the repo shows nothing: cloud build push triggers live in the cloud project, not in `.github/`. A docs-only merge can redeploy production this way. Check all three before calling a merge non-deploy:

1. **Repo:** CI workflow push triggers, their `paths` / `paths-ignore`, and any change-detection job that decides which deploy legs run.
2. **Git host evidence (no cloud access needed):** the check runs from your cloud build app on recent default-branch commits (for GitHub: `gh api repos/<owner>/<repo>/commits/<sha>/check-runs`). `success` with "Build <id> successful" means that push built. `neutral` with "Changed files did not match file filter" means a trigger filter skipped it. No build run at all means no connected trigger fired. Check a commit you know shipped first, as the positive control.
3. **Authoritative:** in the target project, list the build triggers (read `includedFiles` / `ignoredFiles`) and the builds for the commit. This needs build read access; use an admin identity if yours lacks it. **List triggers once per region.** Without a region flag the command lists only global triggers and returns an empty list, with no error, for regional ones. Cover `global` and every region the project deploys in, and run the same command on a project you know has triggers as the positive control before you trust an empty result.

To stop docs-only merges deploying, add `ignoredFiles` to the trigger, but only for paths the image never contains: check the Dockerfile `COPY` list first (some images copy a docs subfolder). Change the trigger once (describe, edit, import); the describe output is the rollback.

## PR merge gate: gate, auto-recover, merge

When a deploy or integration lands through a PR (for example a merge to a `main` that auto-deploys), gate the merge on the check results of the PR's head commit, and merge only that commit. Two scripts bundled with this skill do the gating. `<DEPLOY_SKILL_DIR>` below is this skill's "Base directory for this skill", pasted literally; the scripts need `gh` and `python3`.

- `<DEPLOY_SKILL_DIR>/scripts/pr-checks-green.sh` waits for the PR's checks and prints the head SHA (the only stdout line) only when it is safe to merge that commit. Exit codes: 0 green, 1 red, 2 no checks registered, 3 checks still running at the time limit, 4 usage error, `gh` error, merge conflict or behind.
- `<DEPLOY_SKILL_DIR>/scripts/gate-merge.sh` runs that gate, updates a behind branch (a merge, never a force-push) and gates again up to 3 times, refuses to merge onto a red base branch, merges only the gated commit with `--match-head-commit`, and confirms the PR state is `MERGED`. It is the one-command form of everything in this section.

What "green" means to the gate: every check on the head commit finished with SUCCESS (SKIPPED, CANCELLED and every other result is red); an optional check that came back NEUTRAL (a bot that declined to run) and is not a required check of the base branch counts as passed, while a required NEUTRAL check stays red; every check the base branch requires is present (a job that waits on another job registers its check late, so a list that looks green too early is possible: the gate sees the same head commit green on two polls in a row); every GitHub Actions run for the head commit is complete; the PR has no merge conflict and is not behind a base that requires up-to-date branches.

Run the gate and the merge as ONE background Bash call (the gate waits up to 30 minutes, and `SHA` does not survive into a later call):

```bash
SHA=$(bash "<DEPLOY_SKILL_DIR>/scripts/pr-checks-green.sh" --repo <owner>/<repo> <pr>) \
  && gh pr merge <pr> --repo <owner>/<repo> --squash --match-head-commit "${SHA:?gate did not pass}"
```

or, in the same way, `bash "<DEPLOY_SKILL_DIR>/scripts/gate-merge.sh" <pr> --repo <owner>/<repo>`, which prints `MERGED <pr>` or `STOP: <why>`.

**In a harness that refuses a `$(...)` plus `bash <script>` line** (for example a worktree-bound agent session), split the one-call form into two calls:

1. Run the gate in the background, by the script's literal absolute path (no `bash` in front, no variables), with output to files: `<DEPLOY_SKILL_DIR>/scripts/pr-checks-green.sh --repo <owner>/<repo> <pr> > <scratch>/gate.txt 2> <scratch>/gate.err`
2. When it exits 0, read the head SHA from the first line of `gate.txt` and the `GREEN:` line from `gate.err`, then merge that exact commit: `gh pr merge <pr> --repo <owner>/<repo> --squash --match-head-commit <sha from gate.txt>`.

`--match-head-commit` still refuses the merge if a push moved the head after the gate. Never gate on `gh pr checks --watch`: it can exit 0 on a CANCELLED check, and started right after a push it can finish on the PREVIOUS head's checks.

**Read the PR's inline review comments before every merge, even when branch protection does not require it.** The green gate sees checks only. An automated PR review bot posts its findings as inline review comments, and it can post a blocking finding after CI is green. Where a bot reviews your PRs, wait for it to finish (poll the PR's comments every 60 seconds, for up to 15 minutes, until its summary shows the review completed for the PR's head commit), then read `gh api --paginate repos/<owner>/<repo>/pulls/<pr>/comments` and the PR reviews before `gh pr merge`. Do this again after each push, because a push starts a new review. In such a repo, do not use the one-call forms above: use the split form (gate alone, then wait for and triage the bot's review of the gated commit, then merge with `--match-head-commit`). If the review does not complete in 15 minutes, do not merge on it: run an independent review of the gated commit and merge only when it has no critical or high finding, or ask the owner. Fix a critical or high finding before the merge; reply on the thread for any finding you decline.

**A repo with no CI** (no workflows, no required checks) shows 0 checks on every PR. Without `--allow-none` the gate waits its full 30 minutes, then stops with exit 2, "no checks registered". Add `--allow-none` to the gate line there. It is safe on any repo: the script passes a PR with no checks only after a short grace period, and only when the repo has no workflows and no required checks; anywhere else it still waits for real checks. (A PR with no checks also passes by itself when the base branch demonstrably requires none and the merge state is clean.)

The gate is stricter than the watch, so a non-green result is NOT a stop by itself. Recover ONCE, then gate again:

- **A CANCELLED check, or a known infra flake** (for example a docker build failing on a package-index read timeout): `gh run rerun <run-id> --failed`, then gate again.
- **`BEHIND`:** merge the base branch into the PR branch (`gh pr update-branch`), push, then gate again (the push restarts the checks). `gate-merge.sh` does this itself.
- **The head moved** (`--match-head-commit` refused, or a new push landed): gate again on the new head.
- **Only SKIPPED checks are not SUCCESS, and none of them is a required check of the base branch:** treat it as green and merge with the head SHA (`gh pr view <pr> --json headRefOid`).
- **An optional NEUTRAL check** (for example "review declined" from a bot that is out of credit): the gate already counts a NEUTRAL check that is not a required check of the base branch as passed, and says so in its output. A required check that comes back NEUTRAL stays red.

Stop only when a check still fails after that one recovery pass (a genuine failing test or build). That is a broken change, so report it with the failing job's log excerpt; do not merge around it. Unresolved review threads are the next section.

`gate-merge.sh` also takes `--admin` (self-merge as a ruleset bypass actor; use it only on your own non-deploy PR, and never as the first answer to a BLOCKED merge: see the next section), `--delete-branch` (not from inside a worktree of that repo) and `--allow-red-base` (merge although the base branch is red, only for the PR that fixes it).

## Merge refused as BLOCKED with green CI

When `gh pr merge` refuses and `gh pr view <n> --json mergeStateStatus` prints `BLOCKED` while every required check is green and no approval is required, do this before you reach for an admin override:

1. **Check your repo's own notes first** (a gotchas or runbook doc): another session may already have solved it.
2. **Check for unresolved review threads.** A branch ruleset with `required_review_thread_resolution: true` blocks the merge while any review thread is open, and GitHub's error does not say so. Review bots post inline review comments on PRs, so this recurs:
   ```bash
   gh api repos/<owner>/<repo>/rules/branches/<base> --jq '.[] | select(.type=="pull_request") | .parameters.required_review_thread_resolution'

From itgenius-au/skills-library, MIT.

More from itgenius-au/skills-library