docs(work): archive shipped template epics and PRDs
Move the 8 shipped template epics and their 9 PRDs (incl.
coverage-architecture) to docs/work/archive/{epics,prds}/ so dispatch
context and prioritization only see live Veect work. The state builder
already walks docs/work/epics/ + docs/work/prds/ only; the one work-CLI
script that matched archive paths (bump-updated-timestamps.mjs, staged
docs/work/**/*.md) now excludes docs/work/archive/ so archived content
stays byte-identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016j8z4VHjedXDTjEDNg7qHK
This commit is contained in:
492
docs/work/archive/prds/ci-security-and-supply-chain.prd.md
Normal file
492
docs/work/archive/prds/ci-security-and-supply-chain.prd.md
Normal file
@@ -0,0 +1,492 @@
|
||||
---
|
||||
id: ci-security-and-supply-chain
|
||||
title: CI security + supply-chain enforcement stack
|
||||
type: prd
|
||||
status: approved
|
||||
author: danijel
|
||||
created: 2026-05-14T00:00:00Z
|
||||
updated: 2026-05-14T19:16:52.691Z
|
||||
adr: adr-023
|
||||
builds-on: library-evaluation-policy
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The repo's current security posture, audited 2026-05-14: **zero security tooling**.
|
||||
No Dependabot, no Renovate, no CodeQL, no Snyk, no Trivy, no OSV-Scanner, no
|
||||
Socket, no `gitleaks`, no `pnpm audit signatures` step. GitHub Actions are
|
||||
pinned to **major-version tags** (`actions/checkout@v4`, `pnpm/action-setup@v4`,
|
||||
`googleapis/release-please-action@v4`), which the 2025 `tj-actions/changed-files`
|
||||
incident proved unsafe.
|
||||
|
||||
ADR-022 + the in-flight library-evaluation epic close the **adoption-time** gate
|
||||
for new dependencies. They do not close the **drift** gate. Six post-adoption
|
||||
threats remain uncovered:
|
||||
|
||||
1. **CVE disclosures against pinned versions.** The trace's `verification-commands`
|
||||
snapshot goes stale silently when new advisories drop.
|
||||
2. **Supply-chain _behavior_ compromise** — `event-stream`, `ua-parser-js`,
|
||||
`tj-actions/changed-files`, `xz-utils`. CVE scanning is a lagging indicator;
|
||||
these shipped malware that no CVE database had seen at the moment of compromise.
|
||||
3. **Maintainer-account compromise.** A trusted upstream maintainer's npm
|
||||
account gets phished; the next patch publishes a malicious post-install
|
||||
script; everyone on `^1.2.0` inherits it.
|
||||
4. **GitHub Actions supply chain.** Major-tag pinning is documented insecure.
|
||||
5. **License drift** (e.g. Sentry going BSL on a major; Elasticsearch going SSPL).
|
||||
6. **EU-residency drift** when a vendor announces US-only changes mid-flight.
|
||||
|
||||
ADR-023 codifies the four-pillar enforcement stack that closes these gaps. This
|
||||
PRD implements it.
|
||||
|
||||
## Goal
|
||||
|
||||
A four-pillar CI security stack — Renovate-managed bumps + Action SHA pinning,
|
||||
Socket-based supply-chain-behavior detection, continuous trace revalidation
|
||||
extending ADR-022, and baseline GitHub-native gates — composed via a single
|
||||
failure-mode hierarchy that the sandcastle reviewer prompt enforces machine-readably
|
||||
for agent-driven PRs.
|
||||
|
||||
## In scope
|
||||
|
||||
- **Renovate adoption** — `.github/renovate.json` configuring per-workspace
|
||||
npm bumps (grouped by ecosystem cluster), Dockerfile bumps for
|
||||
`.sandcastle/Dockerfile`, GitHub Actions SHA pinning via
|
||||
`pinGitHubActionDigests`, major/minor split with automerge for green
|
||||
minor+patch PRs.
|
||||
- **One-time Action SHA-pin sweep.** Renovate's first run rewrites the 6
|
||||
existing `uses:` references in `.github/workflows/*.yml` from
|
||||
major-version tags to full 40-char SHAs.
|
||||
- **Socket.dev integration as the 9th hard filter in `evaluate-library`.**
|
||||
Trace schema gains `socket-risk: clean | flagged | "<finding-summary>"`
|
||||
in `filter-results:`. Verification-commands gains the Socket scan command.
|
||||
`.socket.json` configures issue-rules with named severity thresholds
|
||||
(default: `critical → error`).
|
||||
- **Socket CI step** in `ci.yml`'s `validate` job — runs `socket-cli scan`
|
||||
against the lockfile, fails on `critical`.
|
||||
- **Socket GitHub App install instructions** in the human guide for consumers.
|
||||
- **Trace revalidation workflow** at
|
||||
`.github/workflows/trace-revalidation-weekly.yml` — weekly cron +
|
||||
`workflow_dispatch`. Scope: every approved + pre-shipped trace.
|
||||
Two-tier divergence action: soft → rolling dashboard issue
|
||||
(`library-policy/dashboard` label); hard → per-dep issue
|
||||
(`library-policy/re-evaluation` label). No auto-edit of traces, no
|
||||
auto-dispatch, no main-CI gating.
|
||||
- **Trace schema extensions** in `scripts/library-decisions/schema.mjs` —
|
||||
`last-revalidated: <ISO date>` frontmatter field; `socket-risk` filter
|
||||
result; `verification-commands` array gains the Socket entry.
|
||||
- **Major-bump re-evaluation flow** — `scripts/library-decisions/check.mjs`
|
||||
detects when a Renovate PR bumps a runtime dep across a semver-major
|
||||
boundary in a feature/core package and requires the trace's
|
||||
`last-revalidated` to be refreshed. Minor + patch bumps do NOT require
|
||||
re-evaluation.
|
||||
- **CodeQL workflow** at `.github/workflows/codeql.yml` for
|
||||
`javascript-typescript`; runs on push to main + PRs + weekly schedule.
|
||||
- **`pnpm audit signatures --audit-level=high`** added as one step in
|
||||
`ci.yml`'s `validate` job.
|
||||
- **`gitleaks` pre-commit hook** in `.husky/pre-commit` as a step
|
||||
alongside the existing state-sync guard. Custom-pattern allowlist via
|
||||
`.gitleaks.toml`.
|
||||
- **Sandcastle reviewer prompt update** — extend `.sandcastle/reviewer.prompt.md`
|
||||
to read Socket CI output (via `gh run view`) and reject on `critical`,
|
||||
and to read CodeQL findings and reject on `error` severity.
|
||||
- **Failure-mode hierarchy table** ships in
|
||||
`docs/guides/ci-security.md` and is referenced from the reviewer prompt.
|
||||
- **`docs/guides/ci-security.md`** human reading-room — covers each
|
||||
gate, consumer-toggleable settings (GitHub native push protection,
|
||||
Socket App install, branch protection for `library-policy/*` labels),
|
||||
the failure-mode hierarchy table, and worked examples (a passing
|
||||
Renovate minor-bump PR, a blocked major-bump PR, a hard-divergence
|
||||
revalidation issue).
|
||||
- **CLAUDE.md "Key Conventions"** gains a one-line bullet pointing to
|
||||
ADR-023 + the guide.
|
||||
- **Glossary** already includes **Trace revalidation** and
|
||||
**Major-bump re-evaluation** (landed inline during the grill session
|
||||
that produced ADR-023).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Paid Socket Team plan / server-side PR-block enforcement.** The free
|
||||
App + self-hosted CLI achieves equivalent enforcement at $0; paid
|
||||
upgrade is a per-consumer decision.
|
||||
- **Snyk, Trivy, OSV-Scanner.** Free GitHub-native (CodeQL + Dependabot
|
||||
alerts + push protection) + Socket + Renovate cover the surface at $0.
|
||||
- **Container scanning for the sandcastle Dockerfile.** Renovate handles
|
||||
base-image bumps; the sandbox is short-lived and host-isolated.
|
||||
- **Auto-removal of approved-then-unused deps** — `pnpm fallow` territory.
|
||||
- **License auto-enforcement at the lockfile layer** (license-checker
|
||||
plugins). Defer until the policy has run for some time.
|
||||
- **Anything app-tier.** ADR-022 exempts app tier from traces; this PRD
|
||||
inherits that exemption.
|
||||
- **Devdeps in any tier.** Only `dependencies` (runtime) participate in
|
||||
the policy.
|
||||
- **Splitting ADR-023 from ADR-022 amendments into two ADRs.** Decided
|
||||
in the grill (Q7d): one ADR, ADR-022 unedited but cited.
|
||||
- **Auto-dispatch on `library-policy/re-evaluation` issues.** Human
|
||||
triage required; the dispatch loop drains the queue on demand.
|
||||
- **CI gating on `library-policy/re-evaluation` (block main).** Main
|
||||
keeps deploying; trace re-walks happen in parallel.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **ADR-023** is the source of truth. This PRD implements but does not
|
||||
extend it.
|
||||
- **ADR-022** stays unedited. The amendments in ADR-023 §6 are what the
|
||||
implementation honors. Both ADRs must be readable as a composed policy.
|
||||
- **ADR-019** — the sandcastle reviewer prompt is one of four enforcement
|
||||
layers. Reviewer-prompt extensions compose with the existing prompt
|
||||
shape and with the library-evaluation epic's story 06 (whose
|
||||
reviewer-prompt updates land first).
|
||||
- **ADR-021** — release-please picks up dep changes from commit history.
|
||||
Renovate's bump commits must use Conventional Commits (`chore(deps):`,
|
||||
`chore(deps-major):`) so release-please's per-package bump rules apply
|
||||
cleanly.
|
||||
- **Template-vs-consumer framing** — every artifact ships as a
|
||||
consumer-inheritable default. Plan-gated tools (CodeQL on private
|
||||
repos) include clear error messages when the consumer's GitHub plan
|
||||
doesn't cover them.
|
||||
- **Conformance system parity** — the failure-mode hierarchy mirrors
|
||||
ADR-012's latency-tiered shape. Same vocabulary, same agent feedback
|
||||
loop.
|
||||
- **Conventional Commits** — every commit produced by the implementation
|
||||
follows `<type>(<scope>): <subject>`.
|
||||
- **`--no-verify` is forbidden** — the bash-guard hook enforces this; new
|
||||
pre-commit checks inherit the protection.
|
||||
- **Reviewer prompt is the single composable gate for agent PRs** — the
|
||||
sandcastle reviewer must be able to derive "approve/reject" from CI
|
||||
outputs (Socket findings, CodeQL severity) without needing a separate
|
||||
judgment surface.
|
||||
|
||||
## Success criteria
|
||||
|
||||
- `pnpm typecheck && pnpm test && pnpm lint && pnpm conformance && pnpm fallow:audit`
|
||||
pass green at the end of the epic.
|
||||
- `pnpm coverage:diff` covers every changed executable line introduced
|
||||
by the implementation slices.
|
||||
- Renovate's first run (on this repo and on any consumer's downstream
|
||||
fork) opens a PR that SHA-pins every `uses:` reference in
|
||||
`.github/workflows/*.yml`. After merge, no `@v<N>`-style tag pin
|
||||
remains in any workflow.
|
||||
- Running `evaluate-library` against any new package now collects and
|
||||
records a `socket-risk` filter result and includes the `socket-cli`
|
||||
invocation in `verification-commands`. Existing backfilled traces
|
||||
(from the library-evaluation epic) get `socket-risk` added via the
|
||||
Renovate-bump flow or via an explicit backfill task.
|
||||
- A Renovate PR that bumps `@sentry/node` from `7.x → 8.x` against
|
||||
`packages/marketing-pages` is blocked from auto-merge until the
|
||||
trace's `last-revalidated` is refreshed by running the
|
||||
`evaluate-library` skill.
|
||||
- A Renovate PR that bumps `@sentry/node` from `7.5.0 → 7.6.0` against
|
||||
the same package auto-merges if all gates pass — no trace refresh
|
||||
required.
|
||||
- `.github/workflows/trace-revalidation-weekly.yml` runs successfully
|
||||
on its first weekly cron, against the existing ~10 backfilled traces,
|
||||
and produces either zero divergence or a `library-policy/dashboard`
|
||||
issue with a comparison diff.
|
||||
- A simulated hard-divergence trigger (manually mutating a trace's
|
||||
expected `cve-scan` value vs. what `pnpm audit` returns) opens a
|
||||
`library-policy/re-evaluation` issue with the correct title format
|
||||
and body.
|
||||
- A simulated `critical` Socket finding in CI causes the sandcastle
|
||||
reviewer to reject the slice with notes referencing the Socket finding.
|
||||
- A simulated `error`-severity CodeQL finding causes the same reviewer
|
||||
rejection.
|
||||
- `pnpm audit signatures` runs as a step in CI and fails the job when
|
||||
a deliberately-tampered package signature is staged.
|
||||
- `gitleaks` pre-commit hook blocks a commit that adds a known token
|
||||
pattern (Stripe-style test key) to any tracked file.
|
||||
- `docs/guides/ci-security.md` includes the failure-mode hierarchy
|
||||
table, the consumer-toggleable settings list, and at least two
|
||||
worked examples (one approved flow, one blocked flow).
|
||||
- `CLAUDE.md` Key Conventions includes the ADR-023 bullet.
|
||||
- All existing backfilled traces (from the library-evaluation epic)
|
||||
carry a `last-revalidated` field after the first weekly cron run.
|
||||
|
||||
## User stories
|
||||
|
||||
1. **As a developer running `pnpm add` against a feature package**, I want
|
||||
Socket's risk score to be one of the filter results the
|
||||
`evaluate-library` skill collects and records, so I get a single
|
||||
composed answer instead of having to remember to check Socket
|
||||
separately.
|
||||
2. **As an agent dispatched against a slice that bumps a runtime dep**,
|
||||
I want Renovate's bump PR to either auto-merge (minor/patch) or
|
||||
require me to walk `evaluate-library` (major) so the policy gate is
|
||||
automatic, not remembered.
|
||||
3. **As a reviewer (human or agent) of a Renovate major-bump PR**, I
|
||||
want the trace's `last-revalidated` field refreshed by the
|
||||
`evaluate-library` re-run so I can see at a glance "this dep was
|
||||
re-validated today" before approving.
|
||||
4. **As an agent reviewing a slice in sandcastle**, I want the reviewer
|
||||
prompt to read Socket CI output + CodeQL findings + library-trace
|
||||
presence in one composed check and reject on any `critical` /
|
||||
`error`, so I have a single composable gate.
|
||||
5. **As a maintainer who hasn't touched the repo for a week**, I want
|
||||
the weekly trace revalidation cron to produce at most one rolling
|
||||
dashboard issue (soft divergence) and zero per-dep issues (hard
|
||||
divergence) unless something actually drifted, so my notification
|
||||
surface stays clean.
|
||||
6. **As a future agent considering a previously-approved library that's
|
||||
now Socket-flagged**, I want the trace revalidation cron to open a
|
||||
`library-policy/re-evaluation` issue with the trace path + the
|
||||
Socket finding + a clean re-walk handoff, so I can drive the
|
||||
re-evaluation without re-discovering the prior context.
|
||||
7. **As a maintainer reviewing CI output for a PR that touches
|
||||
`package.json`**, I want Socket's comment + the `socket-cli scan`
|
||||
step's result + the library-trace presence check to all be visible
|
||||
in one place (the PR's checks panel), so the decision is one glance,
|
||||
not three.
|
||||
8. **As a security-conscious maintainer**, I want every `uses:`
|
||||
reference in every workflow pinned to a 40-char SHA + a trailing
|
||||
`# v<N>` comment, so the `tj-actions/changed-files` class of attack
|
||||
is closed and Renovate keeps the SHAs current.
|
||||
9. **As a maintainer who accidentally pastes a token into a commit**,
|
||||
I want the `gitleaks` pre-commit hook to refuse the commit + GitHub
|
||||
native push protection to be a second line of defense, so a leaked
|
||||
secret never reaches the remote.
|
||||
10. **As a code reviewer looking at a PR with a CodeQL `error` finding**,
|
||||
I want the finding to appear in PR checks as a hard-block, so the
|
||||
pattern doesn't merge.
|
||||
11. **As a maintainer reading the repo for the first time**, I want
|
||||
`docs/guides/ci-security.md` to walk me through the four pillars
|
||||
- the failure-mode hierarchy + the consumer-toggleable settings,
|
||||
so I understand what to enable in a downstream repo without
|
||||
spelunking workflows.
|
||||
|
||||
## Implementation decisions
|
||||
|
||||
**Module sketch** — what lands where, by concern (no specific file paths
|
||||
where prose suffices):
|
||||
|
||||
- **Renovate config** — single `.github/renovate.json` extending a small
|
||||
set of presets: `config:base`, `helpers:pinGitHubActionDigests`,
|
||||
`:separateMajorReleases`, `:automergeMinor`, `:automergePatch`. Custom
|
||||
`packageRules:` group `@sentry/*`, `@opentelemetry/*`, `@trpc/*`,
|
||||
`payload*`, and `inversify*` into per-cluster weekly PRs. Dockerfile
|
||||
manager enabled for `.sandcastle/Dockerfile`. `dependencyDashboard:
|
||||
true` opens a single issue that summarizes open + queued PRs.
|
||||
- **Socket integration — schema layer.** `scripts/library-decisions/schema.mjs`
|
||||
gains `socketRisk: z.union([z.literal("clean"), z.literal("flagged"),
|
||||
z.string()])` in the `filter-results` Zod schema. The `verification-commands`
|
||||
array gains the Socket entry. The trace template (`_template.md`)
|
||||
mirrors the new field.
|
||||
- **Socket integration — skill layer.** `.claude/skills/evaluate-library/SKILL.md`
|
||||
gains a "9 — Supply-chain behavior (Socket)" section. The skill's
|
||||
fail-fast logic (collect-cheap-skip-expensive) treats Socket as
|
||||
expensive (network call) and runs it after the cheap structural
|
||||
filters. `socket-cli` is the verification command; output parsing
|
||||
follows Socket's JSON schema.
|
||||
- **Socket integration — CI layer.** One step in `ci.yml`'s `validate`
|
||||
job: `socket-cli scan --json | jq <severity-filter>`. Fail on `critical`.
|
||||
`.socket.json` lives at repo root: `{ "issueRules": { "critical":
|
||||
"error", "high": "warn", "medium": "ignore", "low": "ignore" } }`.
|
||||
- **Trace revalidation workflow.** New file
|
||||
`.github/workflows/trace-revalidation-weekly.yml`. Triggers:
|
||||
`schedule: - cron: "30 6 * * 1"` (Monday 06:30 UTC, avoiding the
|
||||
Sunday→Monday CI peak), plus `workflow_dispatch`. Job: checkout,
|
||||
install, run a new script `scripts/library-decisions/revalidate.mjs`
|
||||
that walks every approved + pre-shipped trace, re-runs each trace's
|
||||
`verification-commands`, classifies divergence, opens or updates
|
||||
issues via `gh` CLI. Permissions: `issues: write`, `contents: read`
|
||||
(NO `contents: write` — no auto-edit).
|
||||
- **Major-bump re-evaluation flow.** `scripts/library-decisions/check.mjs`
|
||||
gains a new mode: when invoked on a Renovate-generated PR (detected
|
||||
via branch prefix `renovate/`), it parses the lockfile diff to extract
|
||||
bumped deps + their from/to versions, classifies each as major /
|
||||
minor / patch, and for any feature/core-tier major bump checks that
|
||||
the corresponding trace's `last-revalidated` field is fresh (set
|
||||
today). If not fresh, exit non-zero with a pointer to the
|
||||
`evaluate-library` skill.
|
||||
- **CodeQL workflow.** Standard GitHub-issued template: `language: javascript-typescript`,
|
||||
triggers `push: branches: [main]`, `pull_request`, and weekly
|
||||
`schedule`. Default queries.
|
||||
- **Pre-commit `gitleaks`.** `.husky/pre-commit` gains step:
|
||||
`gitleaks protect --staged --redact`. `.gitleaks.toml` ships with
|
||||
the repo's allowlist patterns (e.g. test fixtures in `__seeds__/`
|
||||
that look like tokens but aren't).
|
||||
- **Reviewer-prompt update.** `.sandcastle/reviewer.prompt.md` gains
|
||||
a "CI security checks" section after the existing library-trace
|
||||
check (from the library-evaluation epic's story 06). The reviewer
|
||||
reads `gh run view` output for the PR's check suite, looks for
|
||||
Socket findings of severity `critical` and CodeQL findings of
|
||||
severity `error`, and rejects the slice if either is present with
|
||||
notes referencing the specific finding.
|
||||
- **Human guide** — `docs/guides/ci-security.md` follows the same
|
||||
shape as `docs/guides/coverage.md`: overview, per-pillar section,
|
||||
failure-mode hierarchy table, consumer settings list, two worked
|
||||
examples.
|
||||
- **CLAUDE.md update** — one bullet in Key Conventions:
|
||||
_"CI security + supply-chain enforcement: Renovate for bumps + Action
|
||||
SHA pinning, Socket for supply-chain behavior, weekly trace
|
||||
revalidation, CodeQL + audit signatures + gitleaks. See ADR-023 +
|
||||
`docs/guides/ci-security.md`."_
|
||||
|
||||
**Trace schema extension (Zod, lifted from ADR-023 §6.3):**
|
||||
|
||||
```ts
|
||||
filterResults: z.object({
|
||||
// ... existing 8 fields ...
|
||||
socketRisk: z.union([
|
||||
z.literal("clean"),
|
||||
z.literal("flagged"),
|
||||
z.string(), // human-readable finding summary
|
||||
]),
|
||||
});
|
||||
lastRevalidated: z.string().nullable(); // ISO date or null on a fresh adoption
|
||||
```
|
||||
|
||||
The `date` field stays mandatory (adoption-provenance); `last-revalidated`
|
||||
is set on major-bump re-eval (Q3) and on a successful trace revalidation
|
||||
run (J).
|
||||
|
||||
**Failure-mode hierarchy (lifted from ADR-023 §5):** the table is the
|
||||
source of truth referenced by both the reviewer prompt and
|
||||
`docs/guides/ci-security.md`. Changes to the hierarchy require an ADR
|
||||
amendment.
|
||||
|
||||
**Sequencing — depends on the library-evaluation epic.** This PRD's
|
||||
implementation depends on the in-flight library-evaluation epic:
|
||||
|
||||
- Story 01 of library-evaluation (trace schema foundation) **must land
|
||||
first** — this PRD extends that schema.
|
||||
- Story 02 of library-evaluation (pre-commit check script) **must land
|
||||
first** — this PRD extends that script with the major-bump-detection
|
||||
mode.
|
||||
- Story 04 of library-evaluation (evaluate-library skill) **must land
|
||||
first** — this PRD adds the Socket filter to that skill.
|
||||
- Story 06 of library-evaluation (reviewer-prompt update) **must land
|
||||
first** — this PRD extends the reviewer prompt added there.
|
||||
|
||||
Sandcastle dispatch should order this PRD's epic _after_ the
|
||||
library-evaluation epic completes.
|
||||
|
||||
**Conformance system composition** — no new use cases, controllers,
|
||||
manifest entries, audits, events, jobs, or realtime channels. This PRD
|
||||
is workflow/policy implementation, not feature-domain change. The
|
||||
conformance gates apply only to the new TypeScript/JS modules (Zod
|
||||
schema extensions, the revalidate.mjs script, the check.mjs major-bump
|
||||
mode) — they get standard vitest coverage.
|
||||
|
||||
## Testing decisions
|
||||
|
||||
- **`scripts/library-decisions/schema.mjs` extensions** — unit tests
|
||||
covering: `socketRisk` field round-trips for all three variants
|
||||
(`clean` / `flagged` / `<string>`); `lastRevalidated` accepts ISO
|
||||
dates and `null`; missing `socketRisk` on a trace fails validation;
|
||||
`lastRevalidated: null` is the default for fresh traces.
|
||||
- **`scripts/library-decisions/check.mjs` major-bump mode** — integration
|
||||
tests covering: minor bump on a feature-tier dep → pass without trace
|
||||
refresh; major bump on a feature-tier dep with fresh `last-revalidated`
|
||||
→ pass; major bump on a feature-tier dep with stale `last-revalidated`
|
||||
→ fail with a clear pointer; major bump on an app-tier dep → pass
|
||||
(app tier exempt); patch bump in a Renovate branch → pass; non-Renovate
|
||||
branch with a major bump → pass (the rule is Renovate-PR-scoped).
|
||||
- **`scripts/library-decisions/revalidate.mjs`** — integration tests
|
||||
using a fixture trace directory: trace with no drift → no issue opened;
|
||||
trace with soft drift → dashboard issue created/updated; trace with
|
||||
hard drift → per-dep issue opened with correct labels + title format;
|
||||
trace already covered by an open per-dep issue → no duplicate issue;
|
||||
rejection trace → skipped entirely. Use `gh` CLI mocks or a fake
|
||||
GitHub API surface for the issue-write side.
|
||||
- **Renovate config** — no automated test; verified by Renovate
|
||||
Dependency Dashboard preview run + manual review of the first PR
|
||||
(the SHA-pin sweep).
|
||||
- **Socket CI step** — smoke test by adding a known-flagged package
|
||||
fixture to a test branch and asserting CI fails. Captures the
|
||||
`socket-cli` output format we depend on for parsing.
|
||||
- **CodeQL workflow** — no test; the workflow file IS the test
|
||||
(GitHub validates the YAML; CodeQL action either runs or no-ops per
|
||||
consumer plan).
|
||||
- **`pnpm audit signatures` step** — verified by the existence of the
|
||||
step in `ci.yml` + a smoke test where a deliberately-corrupt signature
|
||||
fails CI.
|
||||
- **`gitleaks` pre-commit hook** — bash smoke test that pipes a staged
|
||||
commit containing a known token pattern through the hook and asserts
|
||||
exit code non-zero. Use a Stripe-style test key as the fixture.
|
||||
- **Reviewer-prompt extension** — no automated test in the conformance
|
||||
sense (it's a prose runbook for an agent). Success criterion is
|
||||
manual: dispatch an agent against a PR with a simulated Socket
|
||||
`critical` finding, verify the agent rejects with the expected notes.
|
||||
- **Prior art** — mirror the test patterns from the library-evaluation
|
||||
epic's stories 01–02 (trace schema + check script). The fixture and
|
||||
assertion shape carries over directly.
|
||||
- **Coverage bands** — new scripts under `scripts/library-decisions/`
|
||||
aren't feature packages, so no per-layer thresholds. Default expectation:
|
||||
100% statement coverage on the new branches (the scripts are small).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Q1:** Should the Renovate config use a `branchPrefix` other than
|
||||
the default `renovate/` to make the check.mjs Renovate-PR-detection
|
||||
more robust against future Renovate refactors? — **Recommended:**
|
||||
no — Renovate's `renovate/` prefix has been stable for years; use
|
||||
the default and detect via that prefix. Future-proofing here is
|
||||
premature.
|
||||
- **Q2:** Should `socket-cli scan` run on **every** CI PR or only on
|
||||
PRs that touch `package.json` / `pnpm-lock.yaml`? — **Recommended:**
|
||||
only on PRs that touch those files. Use a `paths:` filter on the
|
||||
step. Cheaper CI; same coverage (Socket can't catch behavior changes
|
||||
in a PR that doesn't change deps).
|
||||
- **Q3:** Should the major-bump re-evaluation rule apply when Renovate
|
||||
groups multiple deps in one PR? — **Recommended:** the rule applies
|
||||
per-dep, not per-PR. If a grouped PR contains 3 minor bumps + 1 major
|
||||
bump, the trace for the major-bump dep needs `last-revalidated`
|
||||
refreshed; the 3 minor bumps don't trigger. The check.mjs script
|
||||
walks the lockfile diff and validates each bumped dep independently.
|
||||
- **Q4:** Should `library-policy/re-evaluation` issues auto-close when
|
||||
the trace's `last-revalidated` is refreshed in a subsequent commit?
|
||||
— **Recommended:** yes, the trace revalidation workflow checks for
|
||||
open issues whose dep names appear in newly-refreshed traces and
|
||||
closes them with a comment citing the refresh commit.
|
||||
- **Q5:** Where do the gitleaks allowlist patterns live? — **Recommended:**
|
||||
ship a minimal `.gitleaks.toml` at repo root with one explicit
|
||||
allowlist for `__seeds__/**` test fixtures. Document that consumers
|
||||
extend it for their own custom patterns.
|
||||
- **Q6:** Should the `dependencyDashboard` issue Renovate opens be
|
||||
labeled identically to the trace revalidation `library-policy/dashboard`
|
||||
issue? — **Recommended:** no, keep them separate. Renovate's
|
||||
dependency dashboard is about _pending bumps_; the trace revalidation
|
||||
dashboard is about _post-adoption drift_. Different queues, different
|
||||
labels (`renovate/dashboard` vs `library-policy/dashboard`).
|
||||
|
||||
## Out of scope (deferred)
|
||||
|
||||
- **Branch protection rules.** Configuring GitHub branch protection to
|
||||
require Socket + CodeQL + audit-signatures + library-trace-check
|
||||
checks before merge is a per-repo settings change, not a tracked
|
||||
file. Document the recommended ruleset in
|
||||
`docs/guides/ci-security.md` and leave application to consumers.
|
||||
- **Auto-dispatch on `library-policy/re-evaluation` issues.** Decided
|
||||
out of scope in the grill (Q7c); revisit if human triage becomes a
|
||||
bottleneck in practice.
|
||||
- **Renovate Dependency Dashboard → docs/work/ task integration.**
|
||||
Surfacing pending bumps as `pnpm work` tasks would let agents pick
|
||||
them up via dispatch. Interesting but separate.
|
||||
- **OSSF Scorecard integration.** Complementary to Socket but
|
||||
duplicates several signals; defer until the four-pillar stack has
|
||||
matured.
|
||||
- **StepSecurity Harden Runner.** Adds runtime egress detection on top
|
||||
of Action SHA pinning. Defer; the SHA pins close the primary attack
|
||||
surface.
|
||||
- **Socket Team plan upgrade.** Free tier is documented as adequate
|
||||
for this template; consumers upgrade per their own threat model.
|
||||
- **License-checker lockfile-layer enforcement.** Defer until the
|
||||
policy has run for some time.
|
||||
|
||||
## Further notes
|
||||
|
||||
- **Anchored by ADR-023** — CI security + supply-chain enforcement
|
||||
stack. Read that first.
|
||||
- **Builds on ADR-022** — Library evaluation policy. ADR-022 stays
|
||||
unedited; ADR-023 §6 amends it with major-bump trigger,
|
||||
`last-revalidated` field, and Socket as the 9th hard filter.
|
||||
- **Builds on PRD** `library-evaluation-policy` — many of
|
||||
this PRD's modules extend artifacts being built by that PRD's epic.
|
||||
The sequencing constraint in §Implementation Decisions is
|
||||
load-bearing.
|
||||
- **Glossary entries** for **Trace revalidation** and **Major-bump
|
||||
re-evaluation** landed during the 2026-05-14 grill session that
|
||||
produced ADR-023.
|
||||
- **Conversation provenance** — the 2026-05-14 grill-with-docs session
|
||||
that produced this PRD is captured in the session transcript;
|
||||
ADR-023 cites the audit of zero security tooling + the
|
||||
`tj-actions/changed-files` incident as concrete catalysts.
|
||||
Reference in New Issue
Block a user