agentic-dev-template/docs/guides/ci-security.md

CI security + supply-chain enforcement

Human reading-room for the four-pillar CI security stack. For decision-record density, see ADR-023. For the library evaluation policy that this stack extends, see ADR-022 and docs/guides/adding-a-library.md.

---

Overview: the four pillars

ADR-022 closes the decision gate — every new runtime dependency is evaluated before it enters the lockfile. It does not close the drift gate. Once a library is in the lockfile, ADR-022 has nothing to say about CVEs that surface later, supply-chain compromises against trusted upstream maintainers, license relicensing, or EU-residency changes.

ADR-023 adds four pillars that close the drift gate:

Pillar	Mechanism	Primary signal
1 — Renovate	.github/renovate.json	Keeps lockfile current; SHA-pins GitHub Actions
2 — Socket.dev	GitHub App + socket-cli in CI	Supply-chain behavior — malicious scripts, suspicious network access
3 — Trace revalidation	.github/workflows/trace-revalidation-weekly.yml	CVE drift, license change, EU-residency flip, Socket-flag escalation
4 — GitHub-native gates	CodeQL, push protection, pnpm audit signatures, gitleaks	Static analysis, secret patterns, sigstore provenance

Each pillar enforces at a different latency, mirroring the multi-latency pattern of the conformance system (ADR-012):

pre-commit → CI (per-PR) → server-side (GitHub edge) → weekly cron

No single pillar sees everything. The composition is the enforcement.

---

Pillar 1 — Renovate (bumps + Action SHA pinning)

What it does

Renovate bot manages all dependency bumps via .github/renovate.json. It:

• Opens weekly PRs grouping minor + patch bumps by ecosystem cluster (@sentry/*, @opentelemetry/*, etc.) and auto-merges them when CI is green.
• Opens separate PRs for semver-major bumps. Major bumps do not auto-merge — they block until an agent re-runs evaluate-library and refreshes the trace.
• Rewrites every uses: <owner>/<repo>@<tag> in .github/workflows/*.yml to uses: <owner>/<repo>@<40-char-sha> # <tag>. This closes the tj-actions/changed-files class of supply-chain attack permanently.

What it catches

• CVE patches arriving as minor/patch releases — auto-merged once CI is green.
• License or behavior regressions hidden behind semver-major bumps — blocked until the trace is re-walked.
• Action supply-chain attacks where a compromised maintainer pushes a malicious tag — SHA pins mean the workflow ignores the new tag until Renovate opens a reviewed bump PR.

Toggling in a consumer repo

Renovate requires a GitHub App install (see GitHub Marketplace — Renovate) or a self-hosted Mend Renovate runner. The .github/renovate.json file ships with the template and works unchanged. To customize bump grouping, edit packageRules in that file.

---

Pillar 2 — Socket.dev (supply-chain behavior detection)

What it does

Socket detects supply-chain behavior, not just CVEs. It inspects what a package does — post-install scripts, environment variable access, network calls at install time — not just what a CVE database knows. This catches the event-stream (2018) and ua-parser-js (2021) class of attacks that had no CVE when they shipped.

Two layers:

1. Socket GitHub App — posts a risk-score comment on every PR that touches package.json or pnpm-lock.yaml. Advisory; does not block merge. Free for open-source repos.

2. socket-cli CI step in ci.yml's validate job — runs socket-cli scan against the lockfile. Configured by .socket.json:

   {
     "issueRules": {
       "critical": "error",
       "high": "warn",
       "medium": "ignore",
       "low": "ignore"
     }
   }
   

   critical findings hard-block the PR. high and below are advisory comments only.

Socket is also the 9th hard filter in the evaluate-library skill (ADR-022 adds 8; ADR-023 §6.3 adds this one). Trace frontmatter records the result:

filter-results:
  socket-risk: clean | flagged | "<finding-summary>"

What it catches

• Post-install scripts that write to ~/.ssh or make outbound network calls.
• Packages with maintainer-account-compromise indicators (new maintainer + immediate publish).
• Dependency confusion and typosquat patterns.

Toggling in a consumer repo

The CI step (socket-cli scan) runs unconditionally in CI regardless of App install — no configuration needed beyond the .socket.json that ships with the template. Adjust severity thresholds in .socket.json to match your threat model.

Installing the Socket GitHub App

The App posts PR comments with a risk summary. It's free for public repos and most open-source use.

1. Navigate to socket.dev → Get started → GitHub App.
2. Select the organization or personal account that owns your repo.
3. Grant access to the specific repositories you want covered (or all repositories).
4. The App will start posting on your next PR that touches package.json or pnpm-lock.yaml.

No secrets, no environment variables, and no config file changes are needed — the App reads from your public repo or an authenticated GitHub token.

---

Pillar 3 — Trace revalidation (continuous ADR-022 validation)

What it does

.github/workflows/trace-revalidation-weekly.yml runs every Monday at 06:30 UTC (and on demand via workflow_dispatch). For each approved or pre-shipped trace in docs/library-decisions/:

1. Re-runs the trace's verification-commands block.
2. Compares output against the stored filter-results snapshot.
3. Classifies divergence:
   • Soft — CVE count changed without crossing severity threshold; maintenance signal downgraded one level; transitive dep count changed.
   • Hard — license changed; named consumer no longer present; critical CVE disclosed; EU-residency flipped; Socket flag escalated to critical.

Issue management:

• Soft divergence — appends to a single rolling "library-trace dashboard" issue (labeled library-policy/dashboard), kept open continuously. One issue total; humans skim it periodically; most entries need no action.
• Hard divergence — opens a fresh per-dep issue labeled library-policy/re-evaluation. Title: [trace-revalidation] <package> — <reason>. Body cites the trace path, verification output, and diff.

What revalidation never does:

• Edits a trace file. The re-walk needs the evaluate-library skill (8 filters + 3 prompts, with agent judgement). CI catches divergence; the dispatch loop fixes it.
• Fails CI on main. Main keeps deploying while traces get re-walked. Blocking main on CVE data would stall release-please PRs every time a CVE drops upstream.
• Auto-dispatches library-policy/re-evaluation issues. Issues are a human-triaged queue drained via pnpm work dispatch.

What it catches

• CVEs published against a previously-clean dep.
• License relicensing (MIT → BSL, MIT → SSPL) that ADR-022's adoption-time check didn't see.
• EU-residency changes after a vendor infrastructure announcement.
• Socket flag escalation (package that was clean is now flagged after a maintainer compromise).

Toggling in a consumer repo

The workflow ships in .github/workflows/trace-revalidation-weekly.yml and works without modification. It creates GitHub issues using the GITHUB_TOKEN automatically provided in workflows — no additional secrets needed.

---

Pillar 4 — GitHub-native gates (CodeQL, secret scanning, sigstore, gitleaks)

What it does

Four independent mechanisms in this pillar:

CodeQL (.github/workflows/codeql.yml) — static analysis for JavaScript/TypeScript. Runs on push to main, on PRs, and weekly on Wednesdays. error-severity findings hard-block the PR; warning and note are advisory.

GitHub native push protection — server-side, GitHub edge. Scans for known secret token patterns (API keys, credentials) before accepting a push. Consumer must enable this in repo settings (see Consumer-toggleable settings).

pnpm audit signatures — added as one step in ci.yml's validate job. Verifies npm sigstore attestations. Fails CI when a package in the lockfile has an invalid or missing attestation at --audit-level=high. Roughly 40% of the registry is signed today and the percentage is increasing.

gitleaks pre-commit hook — catches custom secret patterns that GitHub's allowlist doesn't know about (internal API keys, self-hosted service tokens). The hook in .husky/pre-commit runs gitleaks protect --staged --redact and exits gracefully if the gitleaks binary is not installed, so it never blocks developers who haven't set it up yet.

What it catches

• CodeQL — SQL injection, XSS, prototype pollution, command injection, unsafe regex.
• Push protection — AWS keys, GitHub tokens, Stripe keys, and hundreds of other known provider patterns.
• pnpm audit signatures — tampered or unsigned packages in the lockfile.
• gitleaks — custom token patterns, internal service credentials, .env-style secrets accidentally staged.

Toggling in a consumer repo

CodeQL, pnpm audit signatures, and the gitleaks hook all ship with the template and activate without per-consumer config. GitHub native push protection requires a one-time repo settings toggle (see Consumer-toggleable settings).

CodeQL note for private repos

CodeQL is free for public repos and for repos on GitHub Pro/Team/Enterprise plans. Private repos on the free GitHub plan do not have access to GitHub Advanced Security, which CodeQL requires. If you're on a free plan with a private repo, the CodeQL workflow will fail with a clear error message from GitHub rather than silently no-op-ing. Either upgrade your plan or remove the codeql.yml workflow from your fork.

Installing gitleaks for developers

The pre-commit hook exits gracefully if gitleaks is not in PATH — it prints a one-line prompt to install and continues. No developer is blocked by a missing binary. To activate the hook:

macOS (Homebrew):

brew install gitleaks

Linux (apt / package manager):

# Ubuntu/Debian — check the gitleaks GitHub releases for the current version
wget https://github.com/gitleaks/gitleaks/releases/latest/download/gitleaks_linux_amd64.tar.gz
tar -xzf gitleaks_linux_amd64.tar.gz
sudo mv gitleaks /usr/local/bin/

Linux (via go install):

go install github.com/gitleaks/gitleaks/v8@latest

Verify:

gitleaks version

After install, the pre-commit hook runs automatically on git commit. The __seeds__ allowlist in .gitleaks.toml covers intentional test fixtures — add new allowlist entries there if a false positive blocks a commit.

---

Failure-mode hierarchy

Two principles govern what blocks vs. what comments:

• Boolean checks (schema valid, signature verifies, secret present, trace file present) hard-block. They have a definite right answer.
• Judgment checks (Socket risk score, CodeQL semantic finding) are advisory unless severity reaches critical / error. They can have false positives.

Full table — pillar, gate, trigger, action, GitHub label, who resolves:

Pillar	Gate	Trigger	Action	Label	Who resolves
Cross-cutting	pnpm typecheck && test && lint && conformance && coverage:diff	PR / push to main	Hard block PR	—	Developer / agent
Cross-cutting	State-sync guard (pre-commit)	git commit	Block commit	—	Developer / agent
GitHub-native	gitleaks (pre-commit)	git commit	Block commit	—	Developer
Cross-cutting	Library-trace presence check (pre-commit)	git commit	Block commit	—	Developer / agent
GitHub-native	GitHub native push protection	git push	Block push at GitHub edge	—	Developer
Renovate	Minor / patch bump PR	New dep version available	Auto-merge when CI is green	renovate/dashboard	Renovate bot
Renovate	Major bump PR	Semver-major dep version available	Block until evaluate-library re-run + last-revalidated refresh	renovate/dashboard	Agent (via pnpm work dispatch)
Socket	socket-cli CI — critical finding	PR touching package.json / lockfile	Hard block PR	—	Developer / agent
Socket	socket-cli CI — high or below	PR touching package.json / lockfile	Advisory CI annotation	—	Developer (optional)
Socket	Socket GitHub App PR comment	PR touching package.json / lockfile	Advisory comment on PR	—	Developer (optional)
GitHub-native	CodeQL — error severity	Push / PR / weekly schedule	Hard block PR	—	Developer / agent
GitHub-native	CodeQL — warning / note	Push / PR / weekly schedule	Advisory annotation	—	Developer (optional)
GitHub-native	pnpm audit signatures failure	PR / push CI	Hard block PR	—	Developer / agent
GitHub-native	GitHub Dependabot vuln alerts	CVE publication (server-side)	Advisory alert in Security tab	—	Developer (schedule via Renovate bump)
Trace revalidation	Soft divergence	Weekly cron	Append to dashboard issue	library-policy/dashboard	Developer (skim; usually no action)
Trace revalidation	Hard divergence	Weekly cron	Open per-dep issue	library-policy/re-evaluation	Human triage → agent via pnpm work dispatch

---

Consumer-toggleable settings

These three settings are not configured by template files — they require one-time actions in your GitHub repo settings or a third-party install. The template documents them here so consumers know what's available and how to activate each.

GitHub native push protection

Blocks pushes at the GitHub edge if they contain known secret patterns. Free for all plan tiers on public repos; available on GitHub Advanced Security plans for private repos.

To enable: Repo settings → Security → Code security and analysis → Secret scanning → Enable "Push protection".

Once enabled, pushes containing detected patterns are rejected with a link to review the finding. Developers can bypass with a justification if the detection is a false positive.

Socket GitHub App

Adds risk-score comments on PRs that touch package.json or pnpm-lock.yaml. See Installing the Socket GitHub App above.

The CI step (socket-cli scan) runs independently of the App install and hard-blocks on critical findings either way. The App adds the human-readable PR comment layer on top.

Branch protection rules for library-policy/* labels

Optionally, configure branch protection to require human review before merging any PR labeled library-policy/re-evaluation. This prevents an agent loop from auto-closing a re-evaluation issue without a human sign-off.

To enable: Repo settings → Branches → Add rule for main → check "Require approvals" (≥1) → optionally add a status check that verifies the label is resolved before merge.

This is consumer-optional. The template does not enforce it because the right threshold varies by team.

---

Worked examples

Example 1: Passing Renovate minor-bump PR

Scenario: Renovate opens a weekly PR bumping @sentry/node from 8.3.0 to 8.5.0 (a minor bump) and @sentry/nextjs from 8.3.0 to 8.5.0 in the same PR (grouped by the @sentry/* cluster rule in renovate.json).

What happens:

1. Renovate opens the PR — title: chore(deps): bump @sentry/* packages, labeled renovate/dashboard.
2. CI runs the validate job — all five conformance gates pass (typecheck, test, lint, conformance, coverage:diff). pnpm audit signatures passes (no tampered packages). socket-cli scan returns clean (no new install scripts or network behaviors added by the patch).
3. CodeQL workflow runs — no new findings.
4. Socket GitHub App posts a comment: "No new issues found." Advisory; no action required.
5. All checks green → Renovate auto-merges the PR per the automergeMinor: true rule.
6. release-please reads the chore(deps): commit prefix — no version bump (deps bump commits don't trigger a feature-package version). The root template version increments if the commit path is cross-cutting.

No human action required. The full cycle — Renovate opens → CI passes → auto-merge — is self-contained.

---

Example 2: Blocked major-bump PR + hard-divergence revalidation issue

This example covers two related failure modes that appear in the same ecosystem: a Renovate major-bump PR that blocks, and a Socket-flagged hard-divergence issue opened by the weekly cron.

Part A — Blocked major-bump PR

Scenario: Renovate opens a PR bumping zod from 3.22.4 to 4.0.0 (a semver-major bump). The existing trace at docs/library-decisions/2026-05-14-zod.md has last-revalidated: 2026-05-14 (set when Story 05 backfill ran) and version: 3.22.4.

What the pre-commit check does (scripts/library-decisions/check.mjs):

The script detects that zod's version in package.json crosses a semver-major boundary relative to the trace's recorded version field and that last-revalidated predates the bump. The PR comment reads:

[library-trace] zod: semver-major bump detected (3.22.4 → 4.0.0).
Trace last-revalidated: 2026-05-14. Re-run /evaluate-library before merging.

The check does not auto-fail CI — it opens a blocking PR comment. CI itself passes (the code may build fine). But the sandcastle reviewer prompt rejects the slice until the trace is refreshed.

What the agent (or developer) does:

1. Invoke the skill: /evaluate-library zod --tier feature --target packages/auth.
2. The skill re-walks all 9 filters (including Socket scan of zod v4).
3. If zod v4 passes: the trace at docs/library-decisions/2026-05-14-zod.md is updated in-place — version, filter-results, verification-commands, and last-revalidated are refreshed; the original date field is preserved.
4. Commit the updated trace: docs(library-decisions): re-evaluate zod 4.0.0 after major bump.
5. Push to the Renovate PR branch. The check re-runs, finds last-revalidated is current, and unblocks.
6. CI + Socket + CodeQL all pass. The reviewer prompt approves. Renovate's PR merges.

If zod v4 fails a filter (e.g., license changed, Socket finds a new install script), the trace is updated with decision: rejected and the Renovate PR is closed.

Part B — Hard-divergence revalidation issue

Scenario: Two weeks later, socket-cli flags lefthook (a dev dep used in a downstream consumer's fork) with a critical finding after a maintainer-account compromise. The weekly cron runs on Monday morning.

What the cron does:

1. Re-runs socket-cli scan lefthook as part of verification-commands.

2. Compares against the trace's filter-results.socket-risk: clean snapshot.

3. Classifies as hard divergence (Socket flag escalated to critical).

4. Opens a GitHub issue labeled library-policy/re-evaluation:

   Title: [trace-revalidation] lefthook — socket-risk escalated to critical
   Body:
   Trace: docs/library-decisions/2026-05-14-lefthook.md
   Previous socket-risk: clean
   Current socket-risk: critical — malicious post-install script detected
   Verification output: [full socket-cli output pasted here]
   Next step: run /evaluate-library lefthook and update the trace. If rejection is warranted, remove the package.
   

What the developer / agent does:

1. Triage the issue via pnpm work dispatch. The dispatch loop surfaces it as the next ready task.
2. Re-run /evaluate-library lefthook --tier feature --target packages/auth.
3. If the Socket finding is confirmed: decision: rejected, trace updated, pnpm remove lefthook --filter packages/auth, commit.
4. If it's a false positive (Socket has re-classified): socket-risk: clean, trace updated with last-revalidated, issue closed.
5. The agent closes the library-policy/re-evaluation issue with a link to the updated trace commit.

The weekly cron never auto-closes issues. Closing requires an explicit agent or human action — the issue is the queue; the dispatch loop drains it.

---

Cross-links

• ADR-023 — CI security + supply-chain enforcement — the authoritative decision record with full rationale, alternatives considered, and failure-mode hierarchy (§5)
• ADR-022 — Library evaluation policy — the adoption-time gate this stack builds on
• docs/guides/adding-a-library.md — how to add a new runtime dependency (includes the 9-filter evaluation flow)
• .claude/skills/evaluate-library/SKILL.md — agent runbook for evaluate-library
• docs/guides/releasing.md — release-please workflow; how Renovate bump PRs interact with versioning
• docs/glossary.md — entries for Library trace, Pre-shipped trace, Trace revalidation, Major-bump re-evaluation
• ADR-019 — Sandcastle agent orchestration — the reviewer prompt is the agent-loop enforcement surface for Socket + CodeQL gates (ADR-023 §7)