Convention shift: epic folders + PRD filenames + frontmatter id
fields are now bare slugs. The created: timestamp (Phase 2) carries
the date; folder names don't repeat it. A future <task-id>-<slug>
shape (e.g. ClickUp) lands cleanly when that integration ships.
Renames (git mv preserves history):
- docs/work/2026-05-13-binder-wrap-helper/
-> docs/work/binder-wrap-helper/
- docs/work/2026-05-14-library-evaluation-policy/
-> docs/work/library-evaluation-policy/
- docs/work/2026-05-14-ci-security-and-supply-chain/
-> docs/work/ci-security-and-supply-chain/
- docs/work/prds/2026-05-13-binder-wrap-helper.prd.md
-> docs/work/prds/binder-wrap-helper.prd.md
- docs/work/prds/2026-05-13-coverage-architecture.prd.md
-> docs/work/prds/coverage-architecture.prd.md
- docs/work/prds/2026-05-14-library-evaluation-policy.prd.md
-> docs/work/prds/library-evaluation-policy.prd.md
- docs/work/prds/2026-05-14-ci-security-and-supply-chain.prd.md
-> docs/work/prds/ci-security-and-supply-chain.prd.md
Frontmatter updates inside the renamed files: epic id, epic prd,
story epic, PRD id, PRD builds-on all drop date prefixes.
System folder + state file move:
- New docs/work/_system/ holds framework-managed state.
- docs/work/_state.json -> docs/work/_system/_state.json.
- state-builder.mjs adds _system to SKIP_FOLDERS.
- cli.mjs + state-sync-guard.mjs + .husky/pre-commit point at the
new path.
template-reset-v1 epic deleted entirely (one-off cleanup epic from
the pre-date-convention era; status was already done).
Generator-template updates (so new artifacts ship in the right
shape):
- .sandcastle/decomposer.prompt.md emits bare-slug folder names +
ISO created: timestamp.
- .claude/skills/to-prd/SKILL.md template uses bare-slug filename +
bare-slug id field + ISO created: timestamp.
Doc reference updates: glossary, runbook, agent-first-workflow-
and-conformance, reviewer prompt, ADR-020, ADR-022, ADR-023 all
point at the new paths/slugs.
493 lines
26 KiB
Markdown
493 lines
26 KiB
Markdown
---
|
||
id: ci-security-and-supply-chain
|
||
title: CI security + supply-chain enforcement stack
|
||
type: prd
|
||
status: approved
|
||
author: danijel
|
||
created: 2026-05-14T00:00:00Z
|
||
updated: 2026-05-14T19:16:52.691Z
|
||
adr: adr-023
|
||
builds-on: library-evaluation-policy
|
||
---
|
||
|
||
## Problem
|
||
|
||
The repo's current security posture, audited 2026-05-14: **zero security tooling**.
|
||
No Dependabot, no Renovate, no CodeQL, no Snyk, no Trivy, no OSV-Scanner, no
|
||
Socket, no `gitleaks`, no `pnpm audit signatures` step. GitHub Actions are
|
||
pinned to **major-version tags** (`actions/checkout@v4`, `pnpm/action-setup@v4`,
|
||
`googleapis/release-please-action@v4`), which the 2025 `tj-actions/changed-files`
|
||
incident proved unsafe.
|
||
|
||
ADR-022 + the in-flight library-evaluation epic close the **adoption-time** gate
|
||
for new dependencies. They do not close the **drift** gate. Six post-adoption
|
||
threats remain uncovered:
|
||
|
||
1. **CVE disclosures against pinned versions.** The trace's `verification-commands`
|
||
snapshot goes stale silently when new advisories drop.
|
||
2. **Supply-chain _behavior_ compromise** — `event-stream`, `ua-parser-js`,
|
||
`tj-actions/changed-files`, `xz-utils`. CVE scanning is a lagging indicator;
|
||
these shipped malware that no CVE database had seen at the moment of compromise.
|
||
3. **Maintainer-account compromise.** A trusted upstream maintainer's npm
|
||
account gets phished; the next patch publishes a malicious post-install
|
||
script; everyone on `^1.2.0` inherits it.
|
||
4. **GitHub Actions supply chain.** Major-tag pinning is documented insecure.
|
||
5. **License drift** (e.g. Sentry going BSL on a major; Elasticsearch going SSPL).
|
||
6. **EU-residency drift** when a vendor announces US-only changes mid-flight.
|
||
|
||
ADR-023 codifies the four-pillar enforcement stack that closes these gaps. This
|
||
PRD implements it.
|
||
|
||
## Goal
|
||
|
||
A four-pillar CI security stack — Renovate-managed bumps + Action SHA pinning,
|
||
Socket-based supply-chain-behavior detection, continuous trace revalidation
|
||
extending ADR-022, and baseline GitHub-native gates — composed via a single
|
||
failure-mode hierarchy that the sandcastle reviewer prompt enforces machine-readably
|
||
for agent-driven PRs.
|
||
|
||
## In scope
|
||
|
||
- **Renovate adoption** — `.github/renovate.json` configuring per-workspace
|
||
npm bumps (grouped by ecosystem cluster), Dockerfile bumps for
|
||
`.sandcastle/Dockerfile`, GitHub Actions SHA pinning via
|
||
`pinGitHubActionDigests`, major/minor split with automerge for green
|
||
minor+patch PRs.
|
||
- **One-time Action SHA-pin sweep.** Renovate's first run rewrites the 6
|
||
existing `uses:` references in `.github/workflows/*.yml` from
|
||
major-version tags to full 40-char SHAs.
|
||
- **Socket.dev integration as the 9th hard filter in `evaluate-library`.**
|
||
Trace schema gains `socket-risk: clean | flagged | "<finding-summary>"`
|
||
in `filter-results:`. Verification-commands gains the Socket scan command.
|
||
`.socket.json` configures issue-rules with named severity thresholds
|
||
(default: `critical → error`).
|
||
- **Socket CI step** in `ci.yml`'s `validate` job — runs `socket-cli scan`
|
||
against the lockfile, fails on `critical`.
|
||
- **Socket GitHub App install instructions** in the human guide for consumers.
|
||
- **Trace revalidation workflow** at
|
||
`.github/workflows/trace-revalidation-weekly.yml` — weekly cron +
|
||
`workflow_dispatch`. Scope: every approved + pre-shipped trace.
|
||
Two-tier divergence action: soft → rolling dashboard issue
|
||
(`library-policy/dashboard` label); hard → per-dep issue
|
||
(`library-policy/re-evaluation` label). No auto-edit of traces, no
|
||
auto-dispatch, no main-CI gating.
|
||
- **Trace schema extensions** in `scripts/library-decisions/schema.mjs` —
|
||
`last-revalidated: <ISO date>` frontmatter field; `socket-risk` filter
|
||
result; `verification-commands` array gains the Socket entry.
|
||
- **Major-bump re-evaluation flow** — `scripts/library-decisions/check.mjs`
|
||
detects when a Renovate PR bumps a runtime dep across a semver-major
|
||
boundary in a feature/core package and requires the trace's
|
||
`last-revalidated` to be refreshed. Minor + patch bumps do NOT require
|
||
re-evaluation.
|
||
- **CodeQL workflow** at `.github/workflows/codeql.yml` for
|
||
`javascript-typescript`; runs on push to main + PRs + weekly schedule.
|
||
- **`pnpm audit signatures --audit-level=high`** added as one step in
|
||
`ci.yml`'s `validate` job.
|
||
- **`gitleaks` pre-commit hook** in `.husky/pre-commit` as a step
|
||
alongside the existing state-sync guard. Custom-pattern allowlist via
|
||
`.gitleaks.toml`.
|
||
- **Sandcastle reviewer prompt update** — extend `.sandcastle/reviewer.prompt.md`
|
||
to read Socket CI output (via `gh run view`) and reject on `critical`,
|
||
and to read CodeQL findings and reject on `error` severity.
|
||
- **Failure-mode hierarchy table** ships in
|
||
`docs/guides/ci-security.md` and is referenced from the reviewer prompt.
|
||
- **`docs/guides/ci-security.md`** human reading-room — covers each
|
||
gate, consumer-toggleable settings (GitHub native push protection,
|
||
Socket App install, branch protection for `library-policy/*` labels),
|
||
the failure-mode hierarchy table, and worked examples (a passing
|
||
Renovate minor-bump PR, a blocked major-bump PR, a hard-divergence
|
||
revalidation issue).
|
||
- **CLAUDE.md "Key Conventions"** gains a one-line bullet pointing to
|
||
ADR-023 + the guide.
|
||
- **Glossary** already includes **Trace revalidation** and
|
||
**Major-bump re-evaluation** (landed inline during the grill session
|
||
that produced ADR-023).
|
||
|
||
## Out of scope
|
||
|
||
- **Paid Socket Team plan / server-side PR-block enforcement.** The free
|
||
App + self-hosted CLI achieves equivalent enforcement at $0; paid
|
||
upgrade is a per-consumer decision.
|
||
- **Snyk, Trivy, OSV-Scanner.** Free GitHub-native (CodeQL + Dependabot
|
||
alerts + push protection) + Socket + Renovate cover the surface at $0.
|
||
- **Container scanning for the sandcastle Dockerfile.** Renovate handles
|
||
base-image bumps; the sandbox is short-lived and host-isolated.
|
||
- **Auto-removal of approved-then-unused deps** — `pnpm fallow` territory.
|
||
- **License auto-enforcement at the lockfile layer** (license-checker
|
||
plugins). Defer until the policy has run for some time.
|
||
- **Anything app-tier.** ADR-022 exempts app tier from traces; this PRD
|
||
inherits that exemption.
|
||
- **Devdeps in any tier.** Only `dependencies` (runtime) participate in
|
||
the policy.
|
||
- **Splitting ADR-023 from ADR-022 amendments into two ADRs.** Decided
|
||
in the grill (Q7d): one ADR, ADR-022 unedited but cited.
|
||
- **Auto-dispatch on `library-policy/re-evaluation` issues.** Human
|
||
triage required; the dispatch loop drains the queue on demand.
|
||
- **CI gating on `library-policy/re-evaluation` (block main).** Main
|
||
keeps deploying; trace re-walks happen in parallel.
|
||
|
||
## Constraints
|
||
|
||
- **ADR-023** is the source of truth. This PRD implements but does not
|
||
extend it.
|
||
- **ADR-022** stays unedited. The amendments in ADR-023 §6 are what the
|
||
implementation honors. Both ADRs must be readable as a composed policy.
|
||
- **ADR-019** — the sandcastle reviewer prompt is one of four enforcement
|
||
layers. Reviewer-prompt extensions compose with the existing prompt
|
||
shape and with the library-evaluation epic's story 06 (whose
|
||
reviewer-prompt updates land first).
|
||
- **ADR-021** — release-please picks up dep changes from commit history.
|
||
Renovate's bump commits must use Conventional Commits (`chore(deps):`,
|
||
`chore(deps-major):`) so release-please's per-package bump rules apply
|
||
cleanly.
|
||
- **Template-vs-consumer framing** — every artifact ships as a
|
||
consumer-inheritable default. Plan-gated tools (CodeQL on private
|
||
repos) include clear error messages when the consumer's GitHub plan
|
||
doesn't cover them.
|
||
- **Conformance system parity** — the failure-mode hierarchy mirrors
|
||
ADR-012's latency-tiered shape. Same vocabulary, same agent feedback
|
||
loop.
|
||
- **Conventional Commits** — every commit produced by the implementation
|
||
follows `<type>(<scope>): <subject>`.
|
||
- **`--no-verify` is forbidden** — the bash-guard hook enforces this; new
|
||
pre-commit checks inherit the protection.
|
||
- **Reviewer prompt is the single composable gate for agent PRs** — the
|
||
sandcastle reviewer must be able to derive "approve/reject" from CI
|
||
outputs (Socket findings, CodeQL severity) without needing a separate
|
||
judgment surface.
|
||
|
||
## Success criteria
|
||
|
||
- `pnpm typecheck && pnpm test && pnpm lint && pnpm conformance && pnpm fallow:audit`
|
||
pass green at the end of the epic.
|
||
- `pnpm coverage:diff` covers every changed executable line introduced
|
||
by the implementation slices.
|
||
- Renovate's first run (on this repo and on any consumer's downstream
|
||
fork) opens a PR that SHA-pins every `uses:` reference in
|
||
`.github/workflows/*.yml`. After merge, no `@v<N>`-style tag pin
|
||
remains in any workflow.
|
||
- Running `evaluate-library` against any new package now collects and
|
||
records a `socket-risk` filter result and includes the `socket-cli`
|
||
invocation in `verification-commands`. Existing backfilled traces
|
||
(from the library-evaluation epic) get `socket-risk` added via the
|
||
Renovate-bump flow or via an explicit backfill task.
|
||
- A Renovate PR that bumps `@sentry/node` from `7.x → 8.x` against
|
||
`packages/marketing-pages` is blocked from auto-merge until the
|
||
trace's `last-revalidated` is refreshed by running the
|
||
`evaluate-library` skill.
|
||
- A Renovate PR that bumps `@sentry/node` from `7.5.0 → 7.6.0` against
|
||
the same package auto-merges if all gates pass — no trace refresh
|
||
required.
|
||
- `.github/workflows/trace-revalidation-weekly.yml` runs successfully
|
||
on its first weekly cron, against the existing ~10 backfilled traces,
|
||
and produces either zero divergence or a `library-policy/dashboard`
|
||
issue with a comparison diff.
|
||
- A simulated hard-divergence trigger (manually mutating a trace's
|
||
expected `cve-scan` value vs. what `pnpm audit` returns) opens a
|
||
`library-policy/re-evaluation` issue with the correct title format
|
||
and body.
|
||
- A simulated `critical` Socket finding in CI causes the sandcastle
|
||
reviewer to reject the slice with notes referencing the Socket finding.
|
||
- A simulated `error`-severity CodeQL finding causes the same reviewer
|
||
rejection.
|
||
- `pnpm audit signatures` runs as a step in CI and fails the job when
|
||
a deliberately-tampered package signature is staged.
|
||
- `gitleaks` pre-commit hook blocks a commit that adds a known token
|
||
pattern (Stripe-style test key) to any tracked file.
|
||
- `docs/guides/ci-security.md` includes the failure-mode hierarchy
|
||
table, the consumer-toggleable settings list, and at least two
|
||
worked examples (one approved flow, one blocked flow).
|
||
- `CLAUDE.md` Key Conventions includes the ADR-023 bullet.
|
||
- All existing backfilled traces (from the library-evaluation epic)
|
||
carry a `last-revalidated` field after the first weekly cron run.
|
||
|
||
## User stories
|
||
|
||
1. **As a developer running `pnpm add` against a feature package**, I want
|
||
Socket's risk score to be one of the filter results the
|
||
`evaluate-library` skill collects and records, so I get a single
|
||
composed answer instead of having to remember to check Socket
|
||
separately.
|
||
2. **As an agent dispatched against a slice that bumps a runtime dep**,
|
||
I want Renovate's bump PR to either auto-merge (minor/patch) or
|
||
require me to walk `evaluate-library` (major) so the policy gate is
|
||
automatic, not remembered.
|
||
3. **As a reviewer (human or agent) of a Renovate major-bump PR**, I
|
||
want the trace's `last-revalidated` field refreshed by the
|
||
`evaluate-library` re-run so I can see at a glance "this dep was
|
||
re-validated today" before approving.
|
||
4. **As an agent reviewing a slice in sandcastle**, I want the reviewer
|
||
prompt to read Socket CI output + CodeQL findings + library-trace
|
||
presence in one composed check and reject on any `critical` /
|
||
`error`, so I have a single composable gate.
|
||
5. **As a maintainer who hasn't touched the repo for a week**, I want
|
||
the weekly trace revalidation cron to produce at most one rolling
|
||
dashboard issue (soft divergence) and zero per-dep issues (hard
|
||
divergence) unless something actually drifted, so my notification
|
||
surface stays clean.
|
||
6. **As a future agent considering a previously-approved library that's
|
||
now Socket-flagged**, I want the trace revalidation cron to open a
|
||
`library-policy/re-evaluation` issue with the trace path + the
|
||
Socket finding + a clean re-walk handoff, so I can drive the
|
||
re-evaluation without re-discovering the prior context.
|
||
7. **As a maintainer reviewing CI output for a PR that touches
|
||
`package.json`**, I want Socket's comment + the `socket-cli scan`
|
||
step's result + the library-trace presence check to all be visible
|
||
in one place (the PR's checks panel), so the decision is one glance,
|
||
not three.
|
||
8. **As a security-conscious maintainer**, I want every `uses:`
|
||
reference in every workflow pinned to a 40-char SHA + a trailing
|
||
`# v<N>` comment, so the `tj-actions/changed-files` class of attack
|
||
is closed and Renovate keeps the SHAs current.
|
||
9. **As a maintainer who accidentally pastes a token into a commit**,
|
||
I want the `gitleaks` pre-commit hook to refuse the commit + GitHub
|
||
native push protection to be a second line of defense, so a leaked
|
||
secret never reaches the remote.
|
||
10. **As a code reviewer looking at a PR with a CodeQL `error` finding**,
|
||
I want the finding to appear in PR checks as a hard-block, so the
|
||
pattern doesn't merge.
|
||
11. **As a maintainer reading the repo for the first time**, I want
|
||
`docs/guides/ci-security.md` to walk me through the four pillars
|
||
- the failure-mode hierarchy + the consumer-toggleable settings,
|
||
so I understand what to enable in a downstream repo without
|
||
spelunking workflows.
|
||
|
||
## Implementation decisions
|
||
|
||
**Module sketch** — what lands where, by concern (no specific file paths
|
||
where prose suffices):
|
||
|
||
- **Renovate config** — single `.github/renovate.json` extending a small
|
||
set of presets: `config:base`, `helpers:pinGitHubActionDigests`,
|
||
`:separateMajorReleases`, `:automergeMinor`, `:automergePatch`. Custom
|
||
`packageRules:` group `@sentry/*`, `@opentelemetry/*`, `@trpc/*`,
|
||
`payload*`, and `inversify*` into per-cluster weekly PRs. Dockerfile
|
||
manager enabled for `.sandcastle/Dockerfile`. `dependencyDashboard:
|
||
true` opens a single issue that summarizes open + queued PRs.
|
||
- **Socket integration — schema layer.** `scripts/library-decisions/schema.mjs`
|
||
gains `socketRisk: z.union([z.literal("clean"), z.literal("flagged"),
|
||
z.string()])` in the `filter-results` Zod schema. The `verification-commands`
|
||
array gains the Socket entry. The trace template (`_template.md`)
|
||
mirrors the new field.
|
||
- **Socket integration — skill layer.** `.claude/skills/evaluate-library/SKILL.md`
|
||
gains a "9 — Supply-chain behavior (Socket)" section. The skill's
|
||
fail-fast logic (collect-cheap-skip-expensive) treats Socket as
|
||
expensive (network call) and runs it after the cheap structural
|
||
filters. `socket-cli` is the verification command; output parsing
|
||
follows Socket's JSON schema.
|
||
- **Socket integration — CI layer.** One step in `ci.yml`'s `validate`
|
||
job: `socket-cli scan --json | jq <severity-filter>`. Fail on `critical`.
|
||
`.socket.json` lives at repo root: `{ "issueRules": { "critical":
|
||
"error", "high": "warn", "medium": "ignore", "low": "ignore" } }`.
|
||
- **Trace revalidation workflow.** New file
|
||
`.github/workflows/trace-revalidation-weekly.yml`. Triggers:
|
||
`schedule: - cron: "30 6 * * 1"` (Monday 06:30 UTC, avoiding the
|
||
Sunday→Monday CI peak), plus `workflow_dispatch`. Job: checkout,
|
||
install, run a new script `scripts/library-decisions/revalidate.mjs`
|
||
that walks every approved + pre-shipped trace, re-runs each trace's
|
||
`verification-commands`, classifies divergence, opens or updates
|
||
issues via `gh` CLI. Permissions: `issues: write`, `contents: read`
|
||
(NO `contents: write` — no auto-edit).
|
||
- **Major-bump re-evaluation flow.** `scripts/library-decisions/check.mjs`
|
||
gains a new mode: when invoked on a Renovate-generated PR (detected
|
||
via branch prefix `renovate/`), it parses the lockfile diff to extract
|
||
bumped deps + their from/to versions, classifies each as major /
|
||
minor / patch, and for any feature/core-tier major bump checks that
|
||
the corresponding trace's `last-revalidated` field is fresh (set
|
||
today). If not fresh, exit non-zero with a pointer to the
|
||
`evaluate-library` skill.
|
||
- **CodeQL workflow.** Standard GitHub-issued template: `language: javascript-typescript`,
|
||
triggers `push: branches: [main]`, `pull_request`, and weekly
|
||
`schedule`. Default queries.
|
||
- **Pre-commit `gitleaks`.** `.husky/pre-commit` gains step:
|
||
`gitleaks protect --staged --redact`. `.gitleaks.toml` ships with
|
||
the repo's allowlist patterns (e.g. test fixtures in `__seeds__/`
|
||
that look like tokens but aren't).
|
||
- **Reviewer-prompt update.** `.sandcastle/reviewer.prompt.md` gains
|
||
a "CI security checks" section after the existing library-trace
|
||
check (from the library-evaluation epic's story 06). The reviewer
|
||
reads `gh run view` output for the PR's check suite, looks for
|
||
Socket findings of severity `critical` and CodeQL findings of
|
||
severity `error`, and rejects the slice if either is present with
|
||
notes referencing the specific finding.
|
||
- **Human guide** — `docs/guides/ci-security.md` follows the same
|
||
shape as `docs/guides/coverage.md`: overview, per-pillar section,
|
||
failure-mode hierarchy table, consumer settings list, two worked
|
||
examples.
|
||
- **CLAUDE.md update** — one bullet in Key Conventions:
|
||
_"CI security + supply-chain enforcement: Renovate for bumps + Action
|
||
SHA pinning, Socket for supply-chain behavior, weekly trace
|
||
revalidation, CodeQL + audit signatures + gitleaks. See ADR-023 +
|
||
`docs/guides/ci-security.md`."_
|
||
|
||
**Trace schema extension (Zod, lifted from ADR-023 §6.3):**
|
||
|
||
```ts
|
||
filterResults: z.object({
|
||
// ... existing 8 fields ...
|
||
socketRisk: z.union([
|
||
z.literal("clean"),
|
||
z.literal("flagged"),
|
||
z.string(), // human-readable finding summary
|
||
]),
|
||
});
|
||
lastRevalidated: z.string().nullable(); // ISO date or null on a fresh adoption
|
||
```
|
||
|
||
The `date` field stays mandatory (adoption-provenance); `last-revalidated`
|
||
is set on major-bump re-eval (Q3) and on a successful trace revalidation
|
||
run (J).
|
||
|
||
**Failure-mode hierarchy (lifted from ADR-023 §5):** the table is the
|
||
source of truth referenced by both the reviewer prompt and
|
||
`docs/guides/ci-security.md`. Changes to the hierarchy require an ADR
|
||
amendment.
|
||
|
||
**Sequencing — depends on the library-evaluation epic.** This PRD's
|
||
implementation depends on the in-flight library-evaluation epic:
|
||
|
||
- Story 01 of library-evaluation (trace schema foundation) **must land
|
||
first** — this PRD extends that schema.
|
||
- Story 02 of library-evaluation (pre-commit check script) **must land
|
||
first** — this PRD extends that script with the major-bump-detection
|
||
mode.
|
||
- Story 04 of library-evaluation (evaluate-library skill) **must land
|
||
first** — this PRD adds the Socket filter to that skill.
|
||
- Story 06 of library-evaluation (reviewer-prompt update) **must land
|
||
first** — this PRD extends the reviewer prompt added there.
|
||
|
||
Sandcastle dispatch should order this PRD's epic _after_ the
|
||
library-evaluation epic completes.
|
||
|
||
**Conformance system composition** — no new use cases, controllers,
|
||
manifest entries, audits, events, jobs, or realtime channels. This PRD
|
||
is workflow/policy implementation, not feature-domain change. The
|
||
conformance gates apply only to the new TypeScript/JS modules (Zod
|
||
schema extensions, the revalidate.mjs script, the check.mjs major-bump
|
||
mode) — they get standard vitest coverage.
|
||
|
||
## Testing decisions
|
||
|
||
- **`scripts/library-decisions/schema.mjs` extensions** — unit tests
|
||
covering: `socketRisk` field round-trips for all three variants
|
||
(`clean` / `flagged` / `<string>`); `lastRevalidated` accepts ISO
|
||
dates and `null`; missing `socketRisk` on a trace fails validation;
|
||
`lastRevalidated: null` is the default for fresh traces.
|
||
- **`scripts/library-decisions/check.mjs` major-bump mode** — integration
|
||
tests covering: minor bump on a feature-tier dep → pass without trace
|
||
refresh; major bump on a feature-tier dep with fresh `last-revalidated`
|
||
→ pass; major bump on a feature-tier dep with stale `last-revalidated`
|
||
→ fail with a clear pointer; major bump on an app-tier dep → pass
|
||
(app tier exempt); patch bump in a Renovate branch → pass; non-Renovate
|
||
branch with a major bump → pass (the rule is Renovate-PR-scoped).
|
||
- **`scripts/library-decisions/revalidate.mjs`** — integration tests
|
||
using a fixture trace directory: trace with no drift → no issue opened;
|
||
trace with soft drift → dashboard issue created/updated; trace with
|
||
hard drift → per-dep issue opened with correct labels + title format;
|
||
trace already covered by an open per-dep issue → no duplicate issue;
|
||
rejection trace → skipped entirely. Use `gh` CLI mocks or a fake
|
||
GitHub API surface for the issue-write side.
|
||
- **Renovate config** — no automated test; verified by Renovate
|
||
Dependency Dashboard preview run + manual review of the first PR
|
||
(the SHA-pin sweep).
|
||
- **Socket CI step** — smoke test by adding a known-flagged package
|
||
fixture to a test branch and asserting CI fails. Captures the
|
||
`socket-cli` output format we depend on for parsing.
|
||
- **CodeQL workflow** — no test; the workflow file IS the test
|
||
(GitHub validates the YAML; CodeQL action either runs or no-ops per
|
||
consumer plan).
|
||
- **`pnpm audit signatures` step** — verified by the existence of the
|
||
step in `ci.yml` + a smoke test where a deliberately-corrupt signature
|
||
fails CI.
|
||
- **`gitleaks` pre-commit hook** — bash smoke test that pipes a staged
|
||
commit containing a known token pattern through the hook and asserts
|
||
exit code non-zero. Use a Stripe-style test key as the fixture.
|
||
- **Reviewer-prompt extension** — no automated test in the conformance
|
||
sense (it's a prose runbook for an agent). Success criterion is
|
||
manual: dispatch an agent against a PR with a simulated Socket
|
||
`critical` finding, verify the agent rejects with the expected notes.
|
||
- **Prior art** — mirror the test patterns from the library-evaluation
|
||
epic's stories 01–02 (trace schema + check script). The fixture and
|
||
assertion shape carries over directly.
|
||
- **Coverage bands** — new scripts under `scripts/library-decisions/`
|
||
aren't feature packages, so no per-layer thresholds. Default expectation:
|
||
100% statement coverage on the new branches (the scripts are small).
|
||
|
||
## Open questions
|
||
|
||
- **Q1:** Should the Renovate config use a `branchPrefix` other than
|
||
the default `renovate/` to make the check.mjs Renovate-PR-detection
|
||
more robust against future Renovate refactors? — **Recommended:**
|
||
no — Renovate's `renovate/` prefix has been stable for years; use
|
||
the default and detect via that prefix. Future-proofing here is
|
||
premature.
|
||
- **Q2:** Should `socket-cli scan` run on **every** CI PR or only on
|
||
PRs that touch `package.json` / `pnpm-lock.yaml`? — **Recommended:**
|
||
only on PRs that touch those files. Use a `paths:` filter on the
|
||
step. Cheaper CI; same coverage (Socket can't catch behavior changes
|
||
in a PR that doesn't change deps).
|
||
- **Q3:** Should the major-bump re-evaluation rule apply when Renovate
|
||
groups multiple deps in one PR? — **Recommended:** the rule applies
|
||
per-dep, not per-PR. If a grouped PR contains 3 minor bumps + 1 major
|
||
bump, the trace for the major-bump dep needs `last-revalidated`
|
||
refreshed; the 3 minor bumps don't trigger. The check.mjs script
|
||
walks the lockfile diff and validates each bumped dep independently.
|
||
- **Q4:** Should `library-policy/re-evaluation` issues auto-close when
|
||
the trace's `last-revalidated` is refreshed in a subsequent commit?
|
||
— **Recommended:** yes, the trace revalidation workflow checks for
|
||
open issues whose dep names appear in newly-refreshed traces and
|
||
closes them with a comment citing the refresh commit.
|
||
- **Q5:** Where do the gitleaks allowlist patterns live? — **Recommended:**
|
||
ship a minimal `.gitleaks.toml` at repo root with one explicit
|
||
allowlist for `__seeds__/**` test fixtures. Document that consumers
|
||
extend it for their own custom patterns.
|
||
- **Q6:** Should the `dependencyDashboard` issue Renovate opens be
|
||
labeled identically to the trace revalidation `library-policy/dashboard`
|
||
issue? — **Recommended:** no, keep them separate. Renovate's
|
||
dependency dashboard is about _pending bumps_; the trace revalidation
|
||
dashboard is about _post-adoption drift_. Different queues, different
|
||
labels (`renovate/dashboard` vs `library-policy/dashboard`).
|
||
|
||
## Out of scope (deferred)
|
||
|
||
- **Branch protection rules.** Configuring GitHub branch protection to
|
||
require Socket + CodeQL + audit-signatures + library-trace-check
|
||
checks before merge is a per-repo settings change, not a tracked
|
||
file. Document the recommended ruleset in
|
||
`docs/guides/ci-security.md` and leave application to consumers.
|
||
- **Auto-dispatch on `library-policy/re-evaluation` issues.** Decided
|
||
out of scope in the grill (Q7c); revisit if human triage becomes a
|
||
bottleneck in practice.
|
||
- **Renovate Dependency Dashboard → docs/work/ task integration.**
|
||
Surfacing pending bumps as `pnpm work` tasks would let agents pick
|
||
them up via dispatch. Interesting but separate.
|
||
- **OSSF Scorecard integration.** Complementary to Socket but
|
||
duplicates several signals; defer until the four-pillar stack has
|
||
matured.
|
||
- **StepSecurity Harden Runner.** Adds runtime egress detection on top
|
||
of Action SHA pinning. Defer; the SHA pins close the primary attack
|
||
surface.
|
||
- **Socket Team plan upgrade.** Free tier is documented as adequate
|
||
for this template; consumers upgrade per their own threat model.
|
||
- **License-checker lockfile-layer enforcement.** Defer until the
|
||
policy has run for some time.
|
||
|
||
## Further notes
|
||
|
||
- **Anchored by ADR-023** — CI security + supply-chain enforcement
|
||
stack. Read that first.
|
||
- **Builds on ADR-022** — Library evaluation policy. ADR-022 stays
|
||
unedited; ADR-023 §6 amends it with major-bump trigger,
|
||
`last-revalidated` field, and Socket as the 9th hard filter.
|
||
- **Builds on PRD** `library-evaluation-policy` — many of
|
||
this PRD's modules extend artifacts being built by that PRD's epic.
|
||
The sequencing constraint in §Implementation Decisions is
|
||
load-bearing.
|
||
- **Glossary entries** for **Trace revalidation** and **Major-bump
|
||
re-evaluation** landed during the 2026-05-14 grill session that
|
||
produced ADR-023.
|
||
- **Conversation provenance** — the 2026-05-14 grill-with-docs session
|
||
that produced this PRD is captured in the session transcript;
|
||
ADR-023 cites the audit of zero security tooling + the
|
||
`tj-actions/changed-files` incident as concrete catalysts.
|