docs: introduce CI security + supply-chain stack (ADR-023 + PRD)
- ADR-023 codifies the four-pillar enforcement stack: Renovate for bumps + Action SHA pinning via pinGitHubActionDigests, Socket.dev as a 9th hard filter in evaluate-library (free App + self-hosted socket-cli + reviewer-prompt enforcement), weekly trace revalidation cron with two-tier divergence action (rolling dashboard issue + per-dep re-evaluation issues), and the baseline GitHub-native gates (CodeQL, pnpm audit signatures, gitleaks pre-commit + native push protection). Failure-mode hierarchy is the single source of truth referenced by the sandcastle reviewer. - Section 6 amends ADR-022 in place: major-bump re-evaluation trigger (minor/patch bumps skip), last-revalidated frontmatter field (preserves original date for adoption provenance), and Socket as the 9th hard filter. ADR-022 stays unedited; both ADRs read as a composed policy. - PRD at docs/work/prds/2026-05-14-ci-security-and-supply-chain.prd.md seeds the implementation epic; explicit sequencing -- depends on the in-flight library-evaluation epic's stories 01/02/04/06 landing first. - Glossary gains "Trace revalidation" + "Major-bump re-evaluation" entries referenced by both ADRs. Catalyst: 2026-05-14 audit confirmed zero security tooling in the repo + GitHub Actions pinned to major-version tags (the tj-actions/ changed-files attack class). ADR-022 closes the adoption-time gate; ADR-023 closes the post-adoption drift gate.
This commit is contained in:
394
docs/decisions/adr-023-ci-security-and-supply-chain.md
Normal file
394
docs/decisions/adr-023-ci-security-and-supply-chain.md
Normal file
@@ -0,0 +1,394 @@
|
||||
# ADR-023 — CI security + supply-chain enforcement stack
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-05-14
|
||||
**Builds on:** ADR-022 (library evaluation policy)
|
||||
**Related:** ADR-006 (vertical feature packages), ADR-010 (turbo boundaries), ADR-019 (sandcastle agent orchestration), ADR-021 (release-please versioning)
|
||||
**Companion guide:** `docs/guides/ci-security.md` (to be written; human reading-room)
|
||||
|
||||
## Context
|
||||
|
||||
ADR-022 codified the library-evaluation policy: at adoption time, every new
|
||||
runtime dependency in a feature- or core-tier package is gated by 8 hard filters
|
||||
|
||||
- 3 prompts and produces a permanent trace at `docs/library-decisions/`. That
|
||||
closes the **decision** gate. It does not close the **drift** gate. Once a
|
||||
library is in the lockfile, ADR-022 has nothing to say about:
|
||||
|
||||
1. **CVE disclosures against the current pinned version.** A library that
|
||||
passes `pnpm audit --audit-level=moderate` clean at adoption can have a
|
||||
critical CVE published against it the next day. The trace's
|
||||
`verification-commands` snapshot goes stale silently.
|
||||
2. **Supply-chain _behavior_ compromise.** The disclosed-CVE world only
|
||||
catches vulnerabilities that someone has filed. Packages with malicious
|
||||
behavior — `event-stream` (2018), `ua-parser-js` (2021), `tj-actions/changed-files`
|
||||
(2025), `xz-utils` (2024) — shipped malware that no CVE database had
|
||||
seen at the moment of compromise. CVE scanning is a lagging indicator.
|
||||
3. **Maintainer-account compromise.** A trusted upstream maintainer's npm
|
||||
account gets phished. The next `1.2.4` patch publishes a malicious
|
||||
post-install script. Every consumer pulling `^1.2.0` inherits it.
|
||||
Renovate or Dependabot will happily open a bump PR.
|
||||
4. **GitHub Actions supply chain.** This repo's 5 existing workflows pin
|
||||
their actions to **major-version tags** (`actions/checkout@v4`,
|
||||
`pnpm/action-setup@v4`, `googleapis/release-please-action@v4`). The
|
||||
`tj-actions/changed-files` incident demonstrated that a compromised
|
||||
maintainer can push a malicious tag and everyone pinned to `@v4`
|
||||
silently inherits it. Major-tag pinning is documented insecure.
|
||||
5. **License drift.** Upstream packages occasionally relicense (Sentry
|
||||
went BSL on a major; Elasticsearch went SSPL). A `1.x → 2.x` Renovate
|
||||
PR might silently move a previously MIT-licensed dep to a copyleft or
|
||||
source-available license that violates ADR-022's filter #1.
|
||||
6. **EU-residency drift.** A vendor (Sentry, PostHog, etc.) announces
|
||||
US-only changes mid-flight. The trace's `eu-residency: ok` snapshot
|
||||
becomes false; ADR-022's filter #6 has no way to detect this.
|
||||
|
||||
The repo's current security posture, audited 2026-05-14: **zero security
|
||||
tooling**. No Dependabot config, no Renovate, no CodeQL, no Snyk, no Trivy,
|
||||
no OSV-Scanner, no Socket, no gitleaks, no `pnpm audit signatures` step.
|
||||
GitHub Actions are pinned to major-version tags. The 5 existing workflows
|
||||
(`ci.yml`, `coverage-snapshot.yml`, `mutation-nightly.yml`, `release-please.yml`,
|
||||
`sentry-pii-guard.yml`) cover functional CI, but nothing surfaces a
|
||||
post-adoption supply-chain signal.
|
||||
|
||||
For a GDPR-bound EU-resident template that ships as agent-friendly
|
||||
infrastructure, this is the load-bearing gap that ADR-022 cannot close
|
||||
alone. The decision below extends ADR-022 with a continuous-validation
|
||||
counterpart and adds five orthogonal layers that catch the threat surface
|
||||
ADR-022's adoption-time gate doesn't see.
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt a **four-pillar CI security and supply-chain enforcement stack**:
|
||||
(1) Renovate-managed bumps + Action SHA pinning, (2) Socket-based
|
||||
supply-chain-behavior detection, (3) continuous trace revalidation
|
||||
extending ADR-022, (4) baseline GitHub-native gates (CodeQL + secret
|
||||
scanning + sigstore provenance). Each pillar composes with the existing
|
||||
5-gate conformance pattern from ADR-012 — layered enforcement at
|
||||
declining latencies.
|
||||
|
||||
### 1. Renovate adoption (bumps + Action SHA pinning)
|
||||
|
||||
`.github/renovate.json` configures Renovate to manage all runtime + dev
|
||||
dependency bumps and to SHA-pin every GitHub Action invocation:
|
||||
|
||||
- **npm bumps** — per-workspace package.json updates honored. Minor +
|
||||
patch bumps grouped by ecosystem cluster (e.g. one weekly PR for all
|
||||
`@sentry/*`, one for all `@opentelemetry/*`). Auto-merge enabled for
|
||||
green minor + patch PRs. Major bumps require human (or agent) review
|
||||
— see §3.
|
||||
- **Dockerfile bumps** — `.sandcastle/Dockerfile`'s `node:22-bookworm-slim`
|
||||
base image gets the same treatment as npm.
|
||||
- **Action SHA pinning** — `pinGitHubActionDigests` rewrites every
|
||||
`uses: <owner>/<repo>@<tag>` to `uses: <owner>/<repo>@<40-char-sha> # <tag>`
|
||||
on first run. Subsequent Action releases produce bump PRs that update
|
||||
the SHA + the trailing comment in one diff.
|
||||
- **Vulnerability alerts** stay on the GitHub-native server-side surface
|
||||
(no Dependabot bump PRs). Server-side alerts compose with Renovate
|
||||
bumps: an alert may trigger a manual Renovate `:rebase` to accelerate
|
||||
a particular bump.
|
||||
|
||||
Renovate over Dependabot for this repo specifically because:
|
||||
|
||||
- **pnpm-workspace support** is mature and per-workspace updates work
|
||||
from one config file (Dependabot requires verbose per-workspace blocks).
|
||||
- **`pinGitHubActionDigests` is native** (Dependabot SHA-pinning requires
|
||||
manual config).
|
||||
- **PR grouping rules** are more granular — one PR per ecosystem cluster
|
||||
instead of per-package noise.
|
||||
- **Major/minor split + automerge** is one-liner config (Dependabot
|
||||
requires a separate GitHub Action for automerge).
|
||||
|
||||
### 2. Socket.dev integration (supply-chain _behavior_ detection)
|
||||
|
||||
Layered free-tier integration; no paid plan required:
|
||||
|
||||
- **Socket GitHub App** installed on the repo. Posts risk-score comments
|
||||
on every PR that touches `package.json` / `pnpm-lock.yaml`. Free
|
||||
for OSS use.
|
||||
- **`socket-cli` CI step** in `ci.yml`'s `validate` job. Runs
|
||||
`socket-cli scan` against the lockfile and fails the job on
|
||||
configurable severity. Configuration in `.socket.json`:
|
||||
```json
|
||||
{ "issueRules": { "critical": "error", "high": "warn", "medium": "ignore" } }
|
||||
```
|
||||
Default: critical → block the PR; lower severities → comment only.
|
||||
- **Sandcastle reviewer prompt** reads Socket CI output via the GitHub
|
||||
CLI and rejects the agent slice when a `critical` finding is present.
|
||||
Adds machine-readable enforcement to the agent dispatch loop.
|
||||
|
||||
Socket adds a **9th hard filter** to `evaluate-library` (ADR-022's filter
|
||||
set). New trace frontmatter field:
|
||||
|
||||
```yaml
|
||||
filter-results:
|
||||
socket-risk: clean | flagged | "<finding-summary>"
|
||||
```
|
||||
|
||||
At adoption time the skill runs `socket-cli scan <package>` and records
|
||||
the result. The continuous monitor surface is §3 (trace revalidation),
|
||||
which re-runs the same command on schedule.
|
||||
|
||||
### 3. Trace revalidation cron (ADR-022 continuous-validation counterpart)
|
||||
|
||||
New workflow at `.github/workflows/trace-revalidation-weekly.yml`. Runs
|
||||
weekly via cron + on-demand via `workflow_dispatch`. Mirrors the cadence
|
||||
shape of `mutation-nightly.yml`.
|
||||
|
||||
**Scope:** every approved + pre-shipped trace under `docs/library-decisions/`.
|
||||
Rejection traces skipped (no signal value in re-validating an already-rejected
|
||||
library).
|
||||
|
||||
**Action — for each in-scope trace:**
|
||||
|
||||
1. Read the trace's `verification-commands:` block.
|
||||
2. Re-run each command, capture stdout/stderr.
|
||||
3. Compare against the trace's `filter-results:` snapshot.
|
||||
4. Classify divergence:
|
||||
- **Soft** — CVE count changed without crossing severity threshold;
|
||||
maintenance signal still active but downgraded one level; transitive
|
||||
dep count changed.
|
||||
- **Hard** — license changed; named-consumer no longer present;
|
||||
critical CVE disclosed; EU residency flipped to `fail`; Socket
|
||||
flag escalated to `critical`.
|
||||
|
||||
**Issue management:**
|
||||
|
||||
- **Soft divergence** appends to a single rolling **"library-trace
|
||||
dashboard" GitHub issue** kept open continuously. One issue total,
|
||||
updated each run with the latest comparison diff. Labeled
|
||||
`library-policy/dashboard`.
|
||||
- **Hard divergence** opens a fresh per-dep GitHub issue labeled
|
||||
`library-policy/re-evaluation`. Title format:
|
||||
`[trace-revalidation] <package> — <reason>`. The issue body cites the
|
||||
trace path + the verification-command output + the diff.
|
||||
|
||||
**Auto-edit policy:** trace revalidation NEVER edits a trace file. The
|
||||
re-walk needs the `evaluate-library` skill (8 filters + 3 prompts, with
|
||||
agent judgement). CI catches divergence; the dispatch loop fixes it.
|
||||
|
||||
**Auto-dispatch policy:** `library-policy/re-evaluation` issues are
|
||||
**not** auto-picked up by the dispatch loop. Human triage required.
|
||||
Auto-dispatch on CI-opened issues would create a feedback loop where
|
||||
the agent loop spends nights re-evaluating libraries based on rotating
|
||||
CVE data. Issues are a queue; humans drain them via `pnpm work dispatch`.
|
||||
|
||||
**Main-CI gating policy:** hard divergence does NOT fail CI on main.
|
||||
Main can keep deploying while the trace gets re-walked. Gating on main
|
||||
would block release-please PRs every time a CVE drops upstream.
|
||||
|
||||
### 4. Baseline GitHub-native gates
|
||||
|
||||
- **CodeQL** at `.github/workflows/codeql.yml`. Language config
|
||||
`javascript-typescript` covers everything this repo is. Runs on push to
|
||||
main + PRs + weekly schedule. Free for public repos and on
|
||||
Pro/Team/Enterprise plans for private repos; consumers without a
|
||||
CodeQL-eligible plan get a no-op + a clear error message from GitHub.
|
||||
- **`pnpm audit signatures --audit-level=high`** added as one step in
|
||||
`ci.yml`'s existing `validate` job. Verifies npm sigstore attestations.
|
||||
~40% of the registry is signed today and climbing.
|
||||
- **Secret scanning — two layers:**
|
||||
- GitHub-native **push protection** (server-side, free, blocks pushes
|
||||
containing known token patterns at the GitHub edge). Consumer toggles
|
||||
in repo settings. Documented in `docs/guides/ci-security.md`.
|
||||
- **`gitleaks` pre-commit hook** wired into `.husky/pre-commit` as a
|
||||
step alongside the existing state-sync guard. Catches custom token
|
||||
patterns the GitHub allowlist doesn't know about. Local; free.
|
||||
|
||||
### 5. Failure-mode hierarchy
|
||||
|
||||
Two principles govern what blocks vs. what comments:
|
||||
|
||||
- **Boolean checks** (compiles, schema valid, signature verifies, secret
|
||||
present, trace file present) hard-block. They have a definite right
|
||||
answer.
|
||||
- **Judgment checks** (Socket risk score, CodeQL semantic finding) are
|
||||
advisory unless severity reaches `critical` / `error`. They can have
|
||||
false positives.
|
||||
|
||||
Concrete table (the source of truth referenced by reviewer-prompt and
|
||||
documentation):
|
||||
|
||||
| Gate | Layer | Hard block? |
|
||||
| ---------------------------------------------------------------- | ----------- | --------------------------------------------------- |
|
||||
| `pnpm typecheck && test && lint && conformance && coverage:diff` | CI | Yes |
|
||||
| State-sync guard | pre-commit | Yes |
|
||||
| `gitleaks` (custom patterns) | pre-commit | Yes |
|
||||
| Library-trace presence check (ADR-022) | pre-commit | Yes |
|
||||
| GitHub native push protection | server-side | Yes (GitHub edge) |
|
||||
| Renovate minor/patch bump PRs | CI | Auto-merge if green |
|
||||
| Renovate major bump PRs | CI | Block until evaluate-library re-run + trace refresh |
|
||||
| Socket CI step — `critical` | CI | Yes |
|
||||
| Socket CI step — `high` or below | CI | Advisory |
|
||||
| Socket GitHub App PR comments | server-side | Advisory |
|
||||
| CodeQL — `error` severity | CI | Yes |
|
||||
| CodeQL — `warning` / `note` | CI | Advisory |
|
||||
| `pnpm audit signatures` failure | CI | Yes |
|
||||
| GitHub Dependabot vuln alerts | server-side | Advisory (post-merge) |
|
||||
| Trace revalidation — soft divergence | weekly cron | Dashboard issue |
|
||||
| Trace revalidation — hard divergence | weekly cron | Per-dep issue |
|
||||
|
||||
### 6. Amendments to ADR-022
|
||||
|
||||
This ADR amends ADR-022 in three places. ADR-022 itself stays unedited
|
||||
(its `Status: Accepted` is preserved for provenance); the amendments are
|
||||
recorded here and the new behavior is what the implementation honors.
|
||||
|
||||
**§6.1 — Major-bump re-evaluation trigger.** ADR-022 §1 and §8 spoke of
|
||||
"new runtime dependencies" but did not address bumps to existing deps.
|
||||
When Renovate (§1 above) bumps a runtime dep in a feature- or core-tier
|
||||
package and the bump crosses a semver-major boundary, the
|
||||
`evaluate-library` skill re-runs against the upgraded version. Minor +
|
||||
patch bumps do **not** trigger re-evaluation. The existing trace file is
|
||||
updated in-place: `version`, `filter-results`, `verification-commands`,
|
||||
and `last-revalidated` (new field — see §6.2) are refreshed; the original
|
||||
`date` field is preserved as the adoption-provenance marker.
|
||||
|
||||
**§6.2 — `last-revalidated` frontmatter field.** Trace schema gains
|
||||
`last-revalidated: <YYYY-MM-DD>`, set by both major-bump re-eval (§6.1)
|
||||
and trace revalidation (§3). Separate from the original `date` field
|
||||
which is immutable post-adoption.
|
||||
|
||||
**§6.3 — Socket as 9th hard filter.** ADR-022's 8 hard filters gain a
|
||||
9th: `socket-risk`. Trace frontmatter's `filter-results:` block adds
|
||||
`socket-risk: clean | flagged | "<finding-summary>"`. At adoption time
|
||||
the `evaluate-library` skill runs `socket-cli scan <package>` as part of
|
||||
the cheap-structural filter block; `critical` findings auto-reject.
|
||||
Verification-commands gains the Socket scan command.
|
||||
|
||||
### 7. Composition with the sandcastle reviewer prompt
|
||||
|
||||
The reviewer prompt at `.sandcastle/reviewer.prompt.md` is extended with
|
||||
two new responsibilities (bundled into the library-evaluation epic's
|
||||
existing story 06, not split into a separate story):
|
||||
|
||||
- Read Socket CI output (via `gh run view` or PR API) and reject the
|
||||
slice if any `critical` finding is present.
|
||||
- Read CodeQL findings and reject the slice if any `error` severity is
|
||||
present.
|
||||
|
||||
These compose with the reviewer's existing responsibilities (library-
|
||||
trace presence check from ADR-022's PRD, `pnpm coverage:diff` from
|
||||
ADR-020).
|
||||
|
||||
### 8. Template-vs-consumer framing
|
||||
|
||||
This stack ships as **template artifacts** that work in any consumer's
|
||||
GitHub repo. Configurations (`renovate.json`, `.socket.json`, `codeql.yml`,
|
||||
`trace-revalidation-weekly.yml`) are written generically:
|
||||
|
||||
- No project-name-specific paths.
|
||||
- All workflows use `ubuntu-latest`.
|
||||
- Plan-gated tools (CodeQL on private repos) include a clear error
|
||||
message when the consumer's plan doesn't cover them, rather than
|
||||
no-op-ing silently.
|
||||
- `docs/guides/ci-security.md` documents what each consumer toggles
|
||||
(GitHub push protection, Socket App install, branch protection rules
|
||||
for `library-policy/re-evaluation` labels).
|
||||
|
||||
This template's own consumption of the stack — when it's eventually
|
||||
pushed to a GitHub remote — uses the same configurations unchanged.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Dependabot for everything instead of Renovate.** Rejected. Less
|
||||
granular monorepo handling, requires verbose per-workspace config,
|
||||
Action SHA pinning needs manual setup. Renovate's `pnpm-workspace`
|
||||
- `pinGitHubActionDigests` presets do this declaratively.
|
||||
- **No bump tool, manual bumps only.** Rejected. Deps go stale; CVE
|
||||
patches land late; the named-consumer-cares-now signal becomes "who
|
||||
even remembers."
|
||||
- **Paid Socket Team plan for hard PR blocks.** Rejected (default).
|
||||
Free App + self-hosted `socket-cli` in CI achieves equivalent
|
||||
enforcement at $0. Consumers who want the server-side branch-protection
|
||||
integration can upgrade per their own threat model.
|
||||
- **Nightly trace revalidation instead of weekly.** Rejected. License /
|
||||
maintenance / EU-residency signals don't change daily; nightly burns
|
||||
CI minutes on noise. CVE batches publish ~weekly.
|
||||
- **Auto-dispatch on `library-policy/re-evaluation` issues.** Rejected.
|
||||
Creates a feedback loop where the agent loop runs nightly re-evals on
|
||||
rotating CVE data. The issue queue stays human-triaged; the dispatch
|
||||
loop drains it on demand.
|
||||
- **CI gating on `library-policy/re-evaluation` (block main).** Rejected.
|
||||
Main can keep deploying while the trace gets re-walked. CI gating
|
||||
would block release-please PRs every time a CVE drops upstream,
|
||||
conflating release flow with policy maintenance.
|
||||
- **Splitting into two ADRs (CI security + ADR-022 extensions).**
|
||||
Rejected. The bump-trigger rule only makes sense once Renovate is in
|
||||
place; trace revalidation only makes sense alongside Socket; the
|
||||
failure-mode hierarchy spans both. One coherent decision, one ADR.
|
||||
- **Editing ADR-022 in-place to add the bump rule + Socket filter + new
|
||||
field.** Rejected. ADR-022's `Status: Accepted` is provenance — what
|
||||
we believed when it was signed. Amendments live here in §6 and the
|
||||
implementation honors the composed picture. This is the repo's first
|
||||
amendment-style ADR; if it works, future ADR drift gets the same
|
||||
pattern.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- Closes the post-adoption rot ADR-022 cannot reach alone — CVE drift,
|
||||
supply-chain behavior compromise, license drift, EU-residency drift,
|
||||
Action supply-chain attacks.
|
||||
- The trace artifact becomes continuously validated, not point-in-time.
|
||||
Approved libraries carry a `last-revalidated` freshness signal.
|
||||
- Renovate's major/minor split + automerge keeps the lockfile current
|
||||
with minimal human attention while still gating real decision moments.
|
||||
- Action SHA pinning closes the `tj-actions/changed-files` class of
|
||||
attack permanently.
|
||||
- Layered Socket integration (App + CLI + reviewer prompt) gives
|
||||
consumers a $0 baseline that's stricter than most paid offerings'
|
||||
defaults.
|
||||
- Failure-mode hierarchy is machine-readable: the sandcastle reviewer
|
||||
prompt becomes the single composable gate for agent-driven PRs.
|
||||
- Template-vs-consumer framing means downstream repos inherit the
|
||||
stack on day one without per-project setup.
|
||||
|
||||
**Negative**
|
||||
|
||||
- Six new artifacts ship (`renovate.json`, `.socket.json`, `codeql.yml`,
|
||||
`trace-revalidation-weekly.yml`, `gitleaks` pre-commit step, updates
|
||||
to `ci.yml` + reviewer prompt). Each is a maintenance surface.
|
||||
- Renovate config is dense; consumers extending it past defaults need
|
||||
to learn its rules.
|
||||
- Socket GitHub App requires a per-consumer install (one click); the
|
||||
CLI step in CI works regardless.
|
||||
- Trace revalidation produces a steady stream of dashboard-issue
|
||||
updates that humans/agents must skim periodically. Most are
|
||||
no-action.
|
||||
- ADR-022 + ADR-023 together are the repo's first amendment chain.
|
||||
Future agents must read both to get the current policy picture.
|
||||
- Major-bump trigger means every semver-major Renovate PR blocks on
|
||||
agent walk-through of `evaluate-library` (~5 min agent work per
|
||||
major-bump per package).
|
||||
|
||||
**Neutral**
|
||||
|
||||
- The 6 amendments to ADR-022 don't change ADR-022's `Status: Accepted`.
|
||||
Future archaeology finds both ADRs and the implementation honors
|
||||
the composed picture.
|
||||
- Existing 5 workflows are untouched except `ci.yml` gaining 1 step
|
||||
(`pnpm audit signatures`) and 1 step (`socket-cli scan`).
|
||||
- Glossary entries for **Trace revalidation** and **Major-bump
|
||||
re-evaluation** landed inline during the grill session that produced
|
||||
this ADR.
|
||||
|
||||
## Related
|
||||
|
||||
- ADR-022 — Library evaluation policy (the foundation this builds on
|
||||
and amends in §6)
|
||||
- ADR-019 — Sandcastle agent orchestration (the reviewer prompt is the
|
||||
agent-loop enforcement surface, see §7)
|
||||
- ADR-021 — release-please versioning (Renovate's bump PRs interact
|
||||
with release-please's release PRs; both are managed automatically)
|
||||
- ADR-012 — Feature conventions (the conformance system shape this
|
||||
stack mirrors — layered enforcement at declining latencies)
|
||||
- ADR-017 — OpenTelemetry migration (vendor-isolation pattern; Socket
|
||||
integration follows the same shape — `core-shared` doesn't import
|
||||
Socket SDK)
|
||||
- Glossary entries for **Library trace**, **Pre-shipped trace**,
|
||||
**Trace revalidation**, **Major-bump re-evaluation**
|
||||
- PRD: `docs/work/prds/2026-05-14-ci-security-and-supply-chain.prd.md`
|
||||
(to be written, materialized via `/to-prd`)
|
||||
- Companion guide: `docs/guides/ci-security.md` (to be written; human
|
||||
reading-room with worked examples)
|
||||
@@ -302,6 +302,12 @@ A per-decision artifact at `docs/library-decisions/<YYYY-MM-DD>-<package-name>.m
|
||||
**Pre-shipped trace**:
|
||||
A library trace emitted by `pnpm turbo gen core-package <name>` for each direct runtime dep of a pre-curated optional core. Pre-approved by the template's ADRs (015 events / 016 realtime / 018 audit). Generated alongside the core's frozen snapshot so optional cores satisfy the library-evaluation policy by default.
|
||||
|
||||
**Trace revalidation**:
|
||||
The weekly + on-demand CI job (`.github/workflows/trace-revalidation-weekly.yml`) that re-runs every approved trace's `verification-commands` block and detects drift in license / maintenance / CVE / EU-residency / Socket-risk signals that don't show up as version bumps. Two-tier divergence action: **soft divergence** appends to a rolling dashboard issue; **hard divergence** (license changed, named-consumer gone, critical CVE disclosed, EU residency flipped, Socket-flagged) opens a per-dep issue labeled `library-policy/re-evaluation` for the dispatch loop to pick up. Never auto-edits the trace — the re-walk needs the `evaluate-library` skill, not a mechanical edit. Records the most-recent successful pass in the trace's `last-revalidated` frontmatter field (separate from the original adoption `date`).
|
||||
|
||||
**Major-bump re-evaluation**:
|
||||
The Renovate-triggered re-walk of `evaluate-library` when a runtime dep's major version bumps (semver-major, distinct from minor/patch). Updates the existing trace in-place (refreshes `version`, `filter-results`, `verification-commands`, `last-revalidated`), preserves the original `date`. Catches license / maintenance / transitive-surface changes that semver-encodes as "breaking." Minor + patch bumps do not trigger re-evaluation.
|
||||
|
||||
## Modes
|
||||
|
||||
**Production mode**:
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"updated_at": "2026-05-14T10:14:22.260Z",
|
||||
"updated_at": "2026-05-14T16:47:27.076Z",
|
||||
"epics": {
|
||||
"2026-05-13-binder-wrap-helper": {
|
||||
"status": "done",
|
||||
|
||||
491
docs/work/prds/2026-05-14-ci-security-and-supply-chain.prd.md
Normal file
491
docs/work/prds/2026-05-14-ci-security-and-supply-chain.prd.md
Normal file
@@ -0,0 +1,491 @@
|
||||
---
|
||||
id: 2026-05-14-ci-security-and-supply-chain
|
||||
title: CI security + supply-chain enforcement stack
|
||||
type: prd
|
||||
status: appoved
|
||||
author: danijel
|
||||
created: 2026-05-14
|
||||
adr: adr-023
|
||||
builds-on: 2026-05-14-library-evaluation-policy
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The repo's current security posture, audited 2026-05-14: **zero security tooling**.
|
||||
No Dependabot, no Renovate, no CodeQL, no Snyk, no Trivy, no OSV-Scanner, no
|
||||
Socket, no `gitleaks`, no `pnpm audit signatures` step. GitHub Actions are
|
||||
pinned to **major-version tags** (`actions/checkout@v4`, `pnpm/action-setup@v4`,
|
||||
`googleapis/release-please-action@v4`), which the 2025 `tj-actions/changed-files`
|
||||
incident proved unsafe.
|
||||
|
||||
ADR-022 + the in-flight library-evaluation epic close the **adoption-time** gate
|
||||
for new dependencies. They do not close the **drift** gate. Six post-adoption
|
||||
threats remain uncovered:
|
||||
|
||||
1. **CVE disclosures against pinned versions.** The trace's `verification-commands`
|
||||
snapshot goes stale silently when new advisories drop.
|
||||
2. **Supply-chain _behavior_ compromise** — `event-stream`, `ua-parser-js`,
|
||||
`tj-actions/changed-files`, `xz-utils`. CVE scanning is a lagging indicator;
|
||||
these shipped malware that no CVE database had seen at the moment of compromise.
|
||||
3. **Maintainer-account compromise.** A trusted upstream maintainer's npm
|
||||
account gets phished; the next patch publishes a malicious post-install
|
||||
script; everyone on `^1.2.0` inherits it.
|
||||
4. **GitHub Actions supply chain.** Major-tag pinning is documented insecure.
|
||||
5. **License drift** (e.g. Sentry going BSL on a major; Elasticsearch going SSPL).
|
||||
6. **EU-residency drift** when a vendor announces US-only changes mid-flight.
|
||||
|
||||
ADR-023 codifies the four-pillar enforcement stack that closes these gaps. This
|
||||
PRD implements it.
|
||||
|
||||
## Goal
|
||||
|
||||
A four-pillar CI security stack — Renovate-managed bumps + Action SHA pinning,
|
||||
Socket-based supply-chain-behavior detection, continuous trace revalidation
|
||||
extending ADR-022, and baseline GitHub-native gates — composed via a single
|
||||
failure-mode hierarchy that the sandcastle reviewer prompt enforces machine-readably
|
||||
for agent-driven PRs.
|
||||
|
||||
## In scope
|
||||
|
||||
- **Renovate adoption** — `.github/renovate.json` configuring per-workspace
|
||||
npm bumps (grouped by ecosystem cluster), Dockerfile bumps for
|
||||
`.sandcastle/Dockerfile`, GitHub Actions SHA pinning via
|
||||
`pinGitHubActionDigests`, major/minor split with automerge for green
|
||||
minor+patch PRs.
|
||||
- **One-time Action SHA-pin sweep.** Renovate's first run rewrites the 6
|
||||
existing `uses:` references in `.github/workflows/*.yml` from
|
||||
major-version tags to full 40-char SHAs.
|
||||
- **Socket.dev integration as the 9th hard filter in `evaluate-library`.**
|
||||
Trace schema gains `socket-risk: clean | flagged | "<finding-summary>"`
|
||||
in `filter-results:`. Verification-commands gains the Socket scan command.
|
||||
`.socket.json` configures issue-rules with named severity thresholds
|
||||
(default: `critical → error`).
|
||||
- **Socket CI step** in `ci.yml`'s `validate` job — runs `socket-cli scan`
|
||||
against the lockfile, fails on `critical`.
|
||||
- **Socket GitHub App install instructions** in the human guide for consumers.
|
||||
- **Trace revalidation workflow** at
|
||||
`.github/workflows/trace-revalidation-weekly.yml` — weekly cron +
|
||||
`workflow_dispatch`. Scope: every approved + pre-shipped trace.
|
||||
Two-tier divergence action: soft → rolling dashboard issue
|
||||
(`library-policy/dashboard` label); hard → per-dep issue
|
||||
(`library-policy/re-evaluation` label). No auto-edit of traces, no
|
||||
auto-dispatch, no main-CI gating.
|
||||
- **Trace schema extensions** in `scripts/library-decisions/schema.mjs` —
|
||||
`last-revalidated: <ISO date>` frontmatter field; `socket-risk` filter
|
||||
result; `verification-commands` array gains the Socket entry.
|
||||
- **Major-bump re-evaluation flow** — `scripts/library-decisions/check.mjs`
|
||||
detects when a Renovate PR bumps a runtime dep across a semver-major
|
||||
boundary in a feature/core package and requires the trace's
|
||||
`last-revalidated` to be refreshed. Minor + patch bumps do NOT require
|
||||
re-evaluation.
|
||||
- **CodeQL workflow** at `.github/workflows/codeql.yml` for
|
||||
`javascript-typescript`; runs on push to main + PRs + weekly schedule.
|
||||
- **`pnpm audit signatures --audit-level=high`** added as one step in
|
||||
`ci.yml`'s `validate` job.
|
||||
- **`gitleaks` pre-commit hook** in `.husky/pre-commit` as a step
|
||||
alongside the existing state-sync guard. Custom-pattern allowlist via
|
||||
`.gitleaks.toml`.
|
||||
- **Sandcastle reviewer prompt update** — extend `.sandcastle/reviewer.prompt.md`
|
||||
to read Socket CI output (via `gh run view`) and reject on `critical`,
|
||||
and to read CodeQL findings and reject on `error` severity.
|
||||
- **Failure-mode hierarchy table** ships in
|
||||
`docs/guides/ci-security.md` and is referenced from the reviewer prompt.
|
||||
- **`docs/guides/ci-security.md`** human reading-room — covers each
|
||||
gate, consumer-toggleable settings (GitHub native push protection,
|
||||
Socket App install, branch protection for `library-policy/*` labels),
|
||||
the failure-mode hierarchy table, and worked examples (a passing
|
||||
Renovate minor-bump PR, a blocked major-bump PR, a hard-divergence
|
||||
revalidation issue).
|
||||
- **CLAUDE.md "Key Conventions"** gains a one-line bullet pointing to
|
||||
ADR-023 + the guide.
|
||||
- **Glossary** already includes **Trace revalidation** and
|
||||
**Major-bump re-evaluation** (landed inline during the grill session
|
||||
that produced ADR-023).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Paid Socket Team plan / server-side PR-block enforcement.** The free
|
||||
App + self-hosted CLI achieves equivalent enforcement at $0; paid
|
||||
upgrade is a per-consumer decision.
|
||||
- **Snyk, Trivy, OSV-Scanner.** Free GitHub-native (CodeQL + Dependabot
|
||||
alerts + push protection) + Socket + Renovate cover the surface at $0.
|
||||
- **Container scanning for the sandcastle Dockerfile.** Renovate handles
|
||||
base-image bumps; the sandbox is short-lived and host-isolated.
|
||||
- **Auto-removal of approved-then-unused deps** — `pnpm fallow` territory.
|
||||
- **License auto-enforcement at the lockfile layer** (license-checker
|
||||
plugins). Defer until the policy has run for some time.
|
||||
- **Anything app-tier.** ADR-022 exempts app tier from traces; this PRD
|
||||
inherits that exemption.
|
||||
- **Devdeps in any tier.** Only `dependencies` (runtime) participate in
|
||||
the policy.
|
||||
- **Splitting ADR-023 from ADR-022 amendments into two ADRs.** Decided
|
||||
in the grill (Q7d): one ADR, ADR-022 unedited but cited.
|
||||
- **Auto-dispatch on `library-policy/re-evaluation` issues.** Human
|
||||
triage required; the dispatch loop drains the queue on demand.
|
||||
- **CI gating on `library-policy/re-evaluation` (block main).** Main
|
||||
keeps deploying; trace re-walks happen in parallel.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **ADR-023** is the source of truth. This PRD implements but does not
|
||||
extend it.
|
||||
- **ADR-022** stays unedited. The amendments in ADR-023 §6 are what the
|
||||
implementation honors. Both ADRs must be readable as a composed policy.
|
||||
- **ADR-019** — the sandcastle reviewer prompt is one of four enforcement
|
||||
layers. Reviewer-prompt extensions compose with the existing prompt
|
||||
shape and with the library-evaluation epic's story 06 (whose
|
||||
reviewer-prompt updates land first).
|
||||
- **ADR-021** — release-please picks up dep changes from commit history.
|
||||
Renovate's bump commits must use Conventional Commits (`chore(deps):`,
|
||||
`chore(deps-major):`) so release-please's per-package bump rules apply
|
||||
cleanly.
|
||||
- **Template-vs-consumer framing** — every artifact ships as a
|
||||
consumer-inheritable default. Plan-gated tools (CodeQL on private
|
||||
repos) include clear error messages when the consumer's GitHub plan
|
||||
doesn't cover them.
|
||||
- **Conformance system parity** — the failure-mode hierarchy mirrors
|
||||
ADR-012's latency-tiered shape. Same vocabulary, same agent feedback
|
||||
loop.
|
||||
- **Conventional Commits** — every commit produced by the implementation
|
||||
follows `<type>(<scope>): <subject>`.
|
||||
- **`--no-verify` is forbidden** — the bash-guard hook enforces this; new
|
||||
pre-commit checks inherit the protection.
|
||||
- **Reviewer prompt is the single composable gate for agent PRs** — the
|
||||
sandcastle reviewer must be able to derive "approve/reject" from CI
|
||||
outputs (Socket findings, CodeQL severity) without needing a separate
|
||||
judgment surface.
|
||||
|
||||
## Success criteria
|
||||
|
||||
- `pnpm typecheck && pnpm test && pnpm lint && pnpm conformance && pnpm fallow:audit`
|
||||
pass green at the end of the epic.
|
||||
- `pnpm coverage:diff` covers every changed executable line introduced
|
||||
by the implementation slices.
|
||||
- Renovate's first run (on this repo and on any consumer's downstream
|
||||
fork) opens a PR that SHA-pins every `uses:` reference in
|
||||
`.github/workflows/*.yml`. After merge, no `@v<N>`-style tag pin
|
||||
remains in any workflow.
|
||||
- Running `evaluate-library` against any new package now collects and
|
||||
records a `socket-risk` filter result and includes the `socket-cli`
|
||||
invocation in `verification-commands`. Existing backfilled traces
|
||||
(from the library-evaluation epic) get `socket-risk` added via the
|
||||
Renovate-bump flow or via an explicit backfill task.
|
||||
- A Renovate PR that bumps `@sentry/node` from `7.x → 8.x` against
|
||||
`packages/marketing-pages` is blocked from auto-merge until the
|
||||
trace's `last-revalidated` is refreshed by running the
|
||||
`evaluate-library` skill.
|
||||
- A Renovate PR that bumps `@sentry/node` from `7.5.0 → 7.6.0` against
|
||||
the same package auto-merges if all gates pass — no trace refresh
|
||||
required.
|
||||
- `.github/workflows/trace-revalidation-weekly.yml` runs successfully
|
||||
on its first weekly cron, against the existing ~10 backfilled traces,
|
||||
and produces either zero divergence or a `library-policy/dashboard`
|
||||
issue with a comparison diff.
|
||||
- A simulated hard-divergence trigger (manually mutating a trace's
|
||||
expected `cve-scan` value vs. what `pnpm audit` returns) opens a
|
||||
`library-policy/re-evaluation` issue with the correct title format
|
||||
and body.
|
||||
- A simulated `critical` Socket finding in CI causes the sandcastle
|
||||
reviewer to reject the slice with notes referencing the Socket finding.
|
||||
- A simulated `error`-severity CodeQL finding causes the same reviewer
|
||||
rejection.
|
||||
- `pnpm audit signatures` runs as a step in CI and fails the job when
|
||||
a deliberately-tampered package signature is staged.
|
||||
- `gitleaks` pre-commit hook blocks a commit that adds a known token
|
||||
pattern (Stripe-style test key) to any tracked file.
|
||||
- `docs/guides/ci-security.md` includes the failure-mode hierarchy
|
||||
table, the consumer-toggleable settings list, and at least two
|
||||
worked examples (one approved flow, one blocked flow).
|
||||
- `CLAUDE.md` Key Conventions includes the ADR-023 bullet.
|
||||
- All existing backfilled traces (from the library-evaluation epic)
|
||||
carry a `last-revalidated` field after the first weekly cron run.
|
||||
|
||||
## User stories
|
||||
|
||||
1. **As a developer running `pnpm add` against a feature package**, I want
|
||||
Socket's risk score to be one of the filter results the
|
||||
`evaluate-library` skill collects and records, so I get a single
|
||||
composed answer instead of having to remember to check Socket
|
||||
separately.
|
||||
2. **As an agent dispatched against a slice that bumps a runtime dep**,
|
||||
I want Renovate's bump PR to either auto-merge (minor/patch) or
|
||||
require me to walk `evaluate-library` (major) so the policy gate is
|
||||
automatic, not remembered.
|
||||
3. **As a reviewer (human or agent) of a Renovate major-bump PR**, I
|
||||
want the trace's `last-revalidated` field refreshed by the
|
||||
`evaluate-library` re-run so I can see at a glance "this dep was
|
||||
re-validated today" before approving.
|
||||
4. **As an agent reviewing a slice in sandcastle**, I want the reviewer
|
||||
prompt to read Socket CI output + CodeQL findings + library-trace
|
||||
presence in one composed check and reject on any `critical` /
|
||||
`error`, so I have a single composable gate.
|
||||
5. **As a maintainer who hasn't touched the repo for a week**, I want
|
||||
the weekly trace revalidation cron to produce at most one rolling
|
||||
dashboard issue (soft divergence) and zero per-dep issues (hard
|
||||
divergence) unless something actually drifted, so my notification
|
||||
surface stays clean.
|
||||
6. **As a future agent considering a previously-approved library that's
|
||||
now Socket-flagged**, I want the trace revalidation cron to open a
|
||||
`library-policy/re-evaluation` issue with the trace path + the
|
||||
Socket finding + a clean re-walk handoff, so I can drive the
|
||||
re-evaluation without re-discovering the prior context.
|
||||
7. **As a maintainer reviewing CI output for a PR that touches
|
||||
`package.json`**, I want Socket's comment + the `socket-cli scan`
|
||||
step's result + the library-trace presence check to all be visible
|
||||
in one place (the PR's checks panel), so the decision is one glance,
|
||||
not three.
|
||||
8. **As a security-conscious maintainer**, I want every `uses:`
|
||||
reference in every workflow pinned to a 40-char SHA + a trailing
|
||||
`# v<N>` comment, so the `tj-actions/changed-files` class of attack
|
||||
is closed and Renovate keeps the SHAs current.
|
||||
9. **As a maintainer who accidentally pastes a token into a commit**,
|
||||
I want the `gitleaks` pre-commit hook to refuse the commit + GitHub
|
||||
native push protection to be a second line of defense, so a leaked
|
||||
secret never reaches the remote.
|
||||
10. **As a code reviewer looking at a PR with a CodeQL `error` finding**,
|
||||
I want the finding to appear in PR checks as a hard-block, so the
|
||||
pattern doesn't merge.
|
||||
11. **As a maintainer reading the repo for the first time**, I want
|
||||
`docs/guides/ci-security.md` to walk me through the four pillars
|
||||
- the failure-mode hierarchy + the consumer-toggleable settings,
|
||||
so I understand what to enable in a downstream repo without
|
||||
spelunking workflows.
|
||||
|
||||
## Implementation decisions
|
||||
|
||||
**Module sketch** — what lands where, by concern (no specific file paths
|
||||
where prose suffices):
|
||||
|
||||
- **Renovate config** — single `.github/renovate.json` extending a small
|
||||
set of presets: `config:base`, `helpers:pinGitHubActionDigests`,
|
||||
`:separateMajorReleases`, `:automergeMinor`, `:automergePatch`. Custom
|
||||
`packageRules:` group `@sentry/*`, `@opentelemetry/*`, `@trpc/*`,
|
||||
`payload*`, and `inversify*` into per-cluster weekly PRs. Dockerfile
|
||||
manager enabled for `.sandcastle/Dockerfile`. `dependencyDashboard:
|
||||
true` opens a single issue that summarizes open + queued PRs.
|
||||
- **Socket integration — schema layer.** `scripts/library-decisions/schema.mjs`
|
||||
gains `socketRisk: z.union([z.literal("clean"), z.literal("flagged"),
|
||||
z.string()])` in the `filter-results` Zod schema. The `verification-commands`
|
||||
array gains the Socket entry. The trace template (`_template.md`)
|
||||
mirrors the new field.
|
||||
- **Socket integration — skill layer.** `.claude/skills/evaluate-library/SKILL.md`
|
||||
gains a "9 — Supply-chain behavior (Socket)" section. The skill's
|
||||
fail-fast logic (collect-cheap-skip-expensive) treats Socket as
|
||||
expensive (network call) and runs it after the cheap structural
|
||||
filters. `socket-cli` is the verification command; output parsing
|
||||
follows Socket's JSON schema.
|
||||
- **Socket integration — CI layer.** One step in `ci.yml`'s `validate`
|
||||
job: `socket-cli scan --json | jq <severity-filter>`. Fail on `critical`.
|
||||
`.socket.json` lives at repo root: `{ "issueRules": { "critical":
|
||||
"error", "high": "warn", "medium": "ignore", "low": "ignore" } }`.
|
||||
- **Trace revalidation workflow.** New file
|
||||
`.github/workflows/trace-revalidation-weekly.yml`. Triggers:
|
||||
`schedule: - cron: "30 6 * * 1"` (Monday 06:30 UTC, avoiding the
|
||||
Sunday→Monday CI peak), plus `workflow_dispatch`. Job: checkout,
|
||||
install, run a new script `scripts/library-decisions/revalidate.mjs`
|
||||
that walks every approved + pre-shipped trace, re-runs each trace's
|
||||
`verification-commands`, classifies divergence, opens or updates
|
||||
issues via `gh` CLI. Permissions: `issues: write`, `contents: read`
|
||||
(NO `contents: write` — no auto-edit).
|
||||
- **Major-bump re-evaluation flow.** `scripts/library-decisions/check.mjs`
|
||||
gains a new mode: when invoked on a Renovate-generated PR (detected
|
||||
via branch prefix `renovate/`), it parses the lockfile diff to extract
|
||||
bumped deps + their from/to versions, classifies each as major /
|
||||
minor / patch, and for any feature/core-tier major bump checks that
|
||||
the corresponding trace's `last-revalidated` field is fresh (set
|
||||
today). If not fresh, exit non-zero with a pointer to the
|
||||
`evaluate-library` skill.
|
||||
- **CodeQL workflow.** Standard GitHub-issued template: `language: javascript-typescript`,
|
||||
triggers `push: branches: [main]`, `pull_request`, and weekly
|
||||
`schedule`. Default queries.
|
||||
- **Pre-commit `gitleaks`.** `.husky/pre-commit` gains step:
|
||||
`gitleaks protect --staged --redact`. `.gitleaks.toml` ships with
|
||||
the repo's allowlist patterns (e.g. test fixtures in `__seeds__/`
|
||||
that look like tokens but aren't).
|
||||
- **Reviewer-prompt update.** `.sandcastle/reviewer.prompt.md` gains
|
||||
a "CI security checks" section after the existing library-trace
|
||||
check (from the library-evaluation epic's story 06). The reviewer
|
||||
reads `gh run view` output for the PR's check suite, looks for
|
||||
Socket findings of severity `critical` and CodeQL findings of
|
||||
severity `error`, and rejects the slice if either is present with
|
||||
notes referencing the specific finding.
|
||||
- **Human guide** — `docs/guides/ci-security.md` follows the same
|
||||
shape as `docs/guides/coverage.md`: overview, per-pillar section,
|
||||
failure-mode hierarchy table, consumer settings list, two worked
|
||||
examples.
|
||||
- **CLAUDE.md update** — one bullet in Key Conventions:
|
||||
_"CI security + supply-chain enforcement: Renovate for bumps + Action
|
||||
SHA pinning, Socket for supply-chain behavior, weekly trace
|
||||
revalidation, CodeQL + audit signatures + gitleaks. See ADR-023 +
|
||||
`docs/guides/ci-security.md`."_
|
||||
|
||||
**Trace schema extension (Zod, lifted from ADR-023 §6.3):**
|
||||
|
||||
```ts
|
||||
filterResults: z.object({
|
||||
// ... existing 8 fields ...
|
||||
socketRisk: z.union([
|
||||
z.literal("clean"),
|
||||
z.literal("flagged"),
|
||||
z.string(), // human-readable finding summary
|
||||
]),
|
||||
});
|
||||
lastRevalidated: z.string().nullable(); // ISO date or null on a fresh adoption
|
||||
```
|
||||
|
||||
The `date` field stays mandatory (adoption-provenance); `last-revalidated`
|
||||
is set on major-bump re-eval (Q3) and on a successful trace revalidation
|
||||
run (J).
|
||||
|
||||
**Failure-mode hierarchy (lifted from ADR-023 §5):** the table is the
|
||||
source of truth referenced by both the reviewer prompt and
|
||||
`docs/guides/ci-security.md`. Changes to the hierarchy require an ADR
|
||||
amendment.
|
||||
|
||||
**Sequencing — depends on the library-evaluation epic.** This PRD's
|
||||
implementation depends on the in-flight library-evaluation epic:
|
||||
|
||||
- Story 01 of library-evaluation (trace schema foundation) **must land
|
||||
first** — this PRD extends that schema.
|
||||
- Story 02 of library-evaluation (pre-commit check script) **must land
|
||||
first** — this PRD extends that script with the major-bump-detection
|
||||
mode.
|
||||
- Story 04 of library-evaluation (evaluate-library skill) **must land
|
||||
first** — this PRD adds the Socket filter to that skill.
|
||||
- Story 06 of library-evaluation (reviewer-prompt update) **must land
|
||||
first** — this PRD extends the reviewer prompt added there.
|
||||
|
||||
Sandcastle dispatch should order this PRD's epic _after_ the
|
||||
library-evaluation epic completes.
|
||||
|
||||
**Conformance system composition** — no new use cases, controllers,
|
||||
manifest entries, audits, events, jobs, or realtime channels. This PRD
|
||||
is workflow/policy implementation, not feature-domain change. The
|
||||
conformance gates apply only to the new TypeScript/JS modules (Zod
|
||||
schema extensions, the revalidate.mjs script, the check.mjs major-bump
|
||||
mode) — they get standard vitest coverage.
|
||||
|
||||
## Testing decisions
|
||||
|
||||
- **`scripts/library-decisions/schema.mjs` extensions** — unit tests
|
||||
covering: `socketRisk` field round-trips for all three variants
|
||||
(`clean` / `flagged` / `<string>`); `lastRevalidated` accepts ISO
|
||||
dates and `null`; missing `socketRisk` on a trace fails validation;
|
||||
`lastRevalidated: null` is the default for fresh traces.
|
||||
- **`scripts/library-decisions/check.mjs` major-bump mode** — integration
|
||||
tests covering: minor bump on a feature-tier dep → pass without trace
|
||||
refresh; major bump on a feature-tier dep with fresh `last-revalidated`
|
||||
→ pass; major bump on a feature-tier dep with stale `last-revalidated`
|
||||
→ fail with a clear pointer; major bump on an app-tier dep → pass
|
||||
(app tier exempt); patch bump in a Renovate branch → pass; non-Renovate
|
||||
branch with a major bump → pass (the rule is Renovate-PR-scoped).
|
||||
- **`scripts/library-decisions/revalidate.mjs`** — integration tests
|
||||
using a fixture trace directory: trace with no drift → no issue opened;
|
||||
trace with soft drift → dashboard issue created/updated; trace with
|
||||
hard drift → per-dep issue opened with correct labels + title format;
|
||||
trace already covered by an open per-dep issue → no duplicate issue;
|
||||
rejection trace → skipped entirely. Use `gh` CLI mocks or a fake
|
||||
GitHub API surface for the issue-write side.
|
||||
- **Renovate config** — no automated test; verified by Renovate
|
||||
Dependency Dashboard preview run + manual review of the first PR
|
||||
(the SHA-pin sweep).
|
||||
- **Socket CI step** — smoke test by adding a known-flagged package
|
||||
fixture to a test branch and asserting CI fails. Captures the
|
||||
`socket-cli` output format we depend on for parsing.
|
||||
- **CodeQL workflow** — no test; the workflow file IS the test
|
||||
(GitHub validates the YAML; CodeQL action either runs or no-ops per
|
||||
consumer plan).
|
||||
- **`pnpm audit signatures` step** — verified by the existence of the
|
||||
step in `ci.yml` + a smoke test where a deliberately-corrupt signature
|
||||
fails CI.
|
||||
- **`gitleaks` pre-commit hook** — bash smoke test that pipes a staged
|
||||
commit containing a known token pattern through the hook and asserts
|
||||
exit code non-zero. Use a Stripe-style test key as the fixture.
|
||||
- **Reviewer-prompt extension** — no automated test in the conformance
|
||||
sense (it's a prose runbook for an agent). Success criterion is
|
||||
manual: dispatch an agent against a PR with a simulated Socket
|
||||
`critical` finding, verify the agent rejects with the expected notes.
|
||||
- **Prior art** — mirror the test patterns from the library-evaluation
|
||||
epic's stories 01–02 (trace schema + check script). The fixture and
|
||||
assertion shape carries over directly.
|
||||
- **Coverage bands** — new scripts under `scripts/library-decisions/`
|
||||
aren't feature packages, so no per-layer thresholds. Default expectation:
|
||||
100% statement coverage on the new branches (the scripts are small).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Q1:** Should the Renovate config use a `branchPrefix` other than
|
||||
the default `renovate/` to make the check.mjs Renovate-PR-detection
|
||||
more robust against future Renovate refactors? — **Recommended:**
|
||||
no — Renovate's `renovate/` prefix has been stable for years; use
|
||||
the default and detect via that prefix. Future-proofing here is
|
||||
premature.
|
||||
- **Q2:** Should `socket-cli scan` run on **every** CI PR or only on
|
||||
PRs that touch `package.json` / `pnpm-lock.yaml`? — **Recommended:**
|
||||
only on PRs that touch those files. Use a `paths:` filter on the
|
||||
step. Cheaper CI; same coverage (Socket can't catch behavior changes
|
||||
in a PR that doesn't change deps).
|
||||
- **Q3:** Should the major-bump re-evaluation rule apply when Renovate
|
||||
groups multiple deps in one PR? — **Recommended:** the rule applies
|
||||
per-dep, not per-PR. If a grouped PR contains 3 minor bumps + 1 major
|
||||
bump, the trace for the major-bump dep needs `last-revalidated`
|
||||
refreshed; the 3 minor bumps don't trigger. The check.mjs script
|
||||
walks the lockfile diff and validates each bumped dep independently.
|
||||
- **Q4:** Should `library-policy/re-evaluation` issues auto-close when
|
||||
the trace's `last-revalidated` is refreshed in a subsequent commit?
|
||||
— **Recommended:** yes, the trace revalidation workflow checks for
|
||||
open issues whose dep names appear in newly-refreshed traces and
|
||||
closes them with a comment citing the refresh commit.
|
||||
- **Q5:** Where do the gitleaks allowlist patterns live? — **Recommended:**
|
||||
ship a minimal `.gitleaks.toml` at repo root with one explicit
|
||||
allowlist for `__seeds__/**` test fixtures. Document that consumers
|
||||
extend it for their own custom patterns.
|
||||
- **Q6:** Should the `dependencyDashboard` issue Renovate opens be
|
||||
labeled identically to the trace revalidation `library-policy/dashboard`
|
||||
issue? — **Recommended:** no, keep them separate. Renovate's
|
||||
dependency dashboard is about _pending bumps_; the trace revalidation
|
||||
dashboard is about _post-adoption drift_. Different queues, different
|
||||
labels (`renovate/dashboard` vs `library-policy/dashboard`).
|
||||
|
||||
## Out of scope (deferred)
|
||||
|
||||
- **Branch protection rules.** Configuring GitHub branch protection to
|
||||
require Socket + CodeQL + audit-signatures + library-trace-check
|
||||
checks before merge is a per-repo settings change, not a tracked
|
||||
file. Document the recommended ruleset in
|
||||
`docs/guides/ci-security.md` and leave application to consumers.
|
||||
- **Auto-dispatch on `library-policy/re-evaluation` issues.** Decided
|
||||
out of scope in the grill (Q7c); revisit if human triage becomes a
|
||||
bottleneck in practice.
|
||||
- **Renovate Dependency Dashboard → docs/work/ task integration.**
|
||||
Surfacing pending bumps as `pnpm work` tasks would let agents pick
|
||||
them up via dispatch. Interesting but separate.
|
||||
- **OSSF Scorecard integration.** Complementary to Socket but
|
||||
duplicates several signals; defer until the four-pillar stack has
|
||||
matured.
|
||||
- **StepSecurity Harden Runner.** Adds runtime egress detection on top
|
||||
of Action SHA pinning. Defer; the SHA pins close the primary attack
|
||||
surface.
|
||||
- **Socket Team plan upgrade.** Free tier is documented as adequate
|
||||
for this template; consumers upgrade per their own threat model.
|
||||
- **License-checker lockfile-layer enforcement.** Defer until the
|
||||
policy has run for some time.
|
||||
|
||||
## Further notes
|
||||
|
||||
- **Anchored by ADR-023** — CI security + supply-chain enforcement
|
||||
stack. Read that first.
|
||||
- **Builds on ADR-022** — Library evaluation policy. ADR-022 stays
|
||||
unedited; ADR-023 §6 amends it with major-bump trigger,
|
||||
`last-revalidated` field, and Socket as the 9th hard filter.
|
||||
- **Builds on PRD** `2026-05-14-library-evaluation-policy` — many of
|
||||
this PRD's modules extend artifacts being built by that PRD's epic.
|
||||
The sequencing constraint in §Implementation Decisions is
|
||||
load-bearing.
|
||||
- **Glossary entries** for **Trace revalidation** and **Major-bump
|
||||
re-evaluation** landed during the 2026-05-14 grill session that
|
||||
produced ADR-023.
|
||||
- **Conversation provenance** — the 2026-05-14 grill-with-docs session
|
||||
that produced this PRD is captured in the session transcript;
|
||||
ADR-023 cites the audit of zero security tooling + the
|
||||
`tj-actions/changed-files` incident as concrete catalysts.
|
||||
Reference in New Issue
Block a user