Danijel Martinek
432a606411
Add repo-root .socket.json (critical=error, high=warn, medium/low=ignore) and extend the evaluate-library skill's 8-filter protocol to 9 filters by adding Filter 9 — Supply-chain behavior (Socket). Positions Socket last in Phase 2 (most expensive: network call), documents socket-cli as the verification command, and maps clean/flagged/<finding-summary> to the trace's socketRisk field. Updates short-circuit skip lists for all earlier Phase 2 filters and adds socketRisk to TRACE-TEMPLATE.md frontmatter.
215 lines
10 KiB
Markdown
215 lines
10 KiB
Markdown
---
|
|
name: evaluate-library
|
|
description: Walk the 9-filter + 3-prompt library evaluation protocol for a named package, write the decision trace to docs/library-decisions/, and return pass/fail. Use when adding a runtime dependency to a feature or core package, or when the library-policy-nudge hook fires.
|
|
---
|
|
|
|
<invocation>
|
|
|
|
```
|
|
/evaluate-library <package-name> --tier <feature|core|app> --target <package-path>
|
|
```
|
|
|
|
All three arguments are required. The `library-policy-nudge` hook emits this exact invocation. For `app`-tier packages, evaluation still runs but a trace is optional (author's call per ADR-022 §1).
|
|
|
|
</invocation>
|
|
|
|
<runbook>
|
|
|
|
## Overview
|
|
|
|
Walk nine hard auto-reject filters in **collect-cheap-skip-expensive** order, then answer three discussion prompts. Write the trace unconditionally at the end — including for rejections. A rejection trace is a permanent record that prevents future agents from re-litigating the same decision.
|
|
|
|
## Phase 1 — Cheap filters (always run to completion, even if one fails)
|
|
|
|
Run all four cheap filters regardless of their outcomes. Record each result before moving to Phase 2.
|
|
|
|
### Filter 1: license
|
|
|
|
Command: `node -e "const p = JSON.parse(require('fs').readFileSync('./node_modules/<pkg>/package.json','utf8')); console.log(p.license)"`
|
|
|
|
Allowlist: `MIT`, `Apache-2.0`, `BSD-2-Clause`, `BSD-3-Clause`, `ISC`, `MPL-2.0`.
|
|
|
|
Result values: the SPDX identifier (e.g. `MIT`) if allowed, or `<SPDX-id> (rejected)` if outside the allowlist. Anything outside the allowlist is an automatic reject but does not stop Phase 1.
|
|
|
|
### Filter 2: types
|
|
|
|
Check whether TypeScript types ship with the package or via `@types/<pkg>`:
|
|
|
|
```
|
|
ls node_modules/<pkg>/index.d.ts 2>/dev/null && echo native || npm info @types/<pkg> version 2>/dev/null | head -1
|
|
```
|
|
|
|
Result values: `native` (ships its own `.d.ts`), `@types/<pkg>` (community types available), or `none` (auto-reject — un-typed library shifts maintenance cost to the feature).
|
|
|
|
### Filter 3: shadow-check
|
|
|
|
Check whether this library duplicates a must-have already locked in the workspace. Locked must-haves: `zod` (validation), `inversify` (DI, ADR-002), `payload` (CMS), `@trpc/server` (API layer), `superjson` (serialisation), `reflect-metadata` (DI metadata).
|
|
|
|
Command: `cat package.json | grep -E '"(zod|inversify|payload|@trpc/server|superjson|reflect-metadata)"'` — run from the workspace root.
|
|
|
|
Result values: `pass` (no shadow), `fail` (exact duplicate of a locked dep), `"shadows <x>"` (functional parallel that would create two libraries doing the same job — auto-reject). A replacement must be a separate ADR with consequences analysis, not a parallel adoption.
|
|
|
|
### Filter 4: boundary-fit
|
|
|
|
Confirm the dependency does not violate ESLint boundary-tag rules for the target tier (ADR-006, ADR-010, ADR-017).
|
|
|
|
Key rules:
|
|
|
|
- Feature packages cannot import `@sentry/*` or `@opentelemetry/sdk-*` directly — those are reserved for core (ADR-017 §4).
|
|
- No package may import across feature boundaries without going through the event bus or tRPC.
|
|
- Optional core packages can only be imported by apps and `core-composition`-tagged packages.
|
|
|
|
Check by reviewing what the proposed library's transitive imports would bring in and whether any violate the boundary ruleset.
|
|
|
|
Result values: `pass` or `fail`.
|
|
|
|
---
|
|
|
|
After Phase 1: tally results. If **any cheap filter failed**, the overall decision is `rejected`. Proceed to Phase 2 anyway — all expensive filters still run if the Phase 1 decision is already rejected (they inform the full record). If all cheap filters passed, proceed to Phase 2 to determine the final decision.
|
|
|
|
## Phase 2 — Expensive filters (short-circuit after first reject)
|
|
|
|
Run in order. On the first failure, set remaining filter results to `skip` and skip to the [Trace write step](#trace-write-step).
|
|
|
|
### Filter 5: maintenance
|
|
|
|
Check last release date and recent PR/issue activity:
|
|
|
|
```
|
|
npm info <pkg> time.modified
|
|
npm info <pkg> time | tail -5
|
|
```
|
|
|
|
Result values:
|
|
|
|
- `active` — last release < 18 months **and** PR/issue activity < 12 months
|
|
- `dormant` — stable, not actively developed (acceptable for finished libraries like `reflect-metadata`)
|
|
- `abandoned` — last release ≥ 18 months **or** no activity in ≥ 12 months → auto-reject; short-circuit remaining expensive filters
|
|
|
|
On `abandoned` → set `cve-scan`, `eu-residency`, `named-consumer`, `socketRisk` to `skip` → write trace.
|
|
|
|
### Filter 6: cve-scan
|
|
|
|
```
|
|
pnpm audit --audit-level=moderate 2>&1 | head -40
|
|
```
|
|
|
|
Result values: `clean` (no advisories), an advisory ID like `GHSA-xxxx-xxxx-xxxx` (accepted risk — document in `accepted-cves` frontmatter), or `fail` (open advisory not accepted → auto-reject; short-circuit remaining expensive filters).
|
|
|
|
On `fail` → set `eu-residency`, `named-consumer`, `socketRisk` to `skip` → write trace.
|
|
|
|
### Filter 7: eu-residency
|
|
|
|
Applies only if the library transmits user data, telemetry, business state, or secrets to a vendor-controlled endpoint by default. Examples: analytics SDKs, error-tracking clients, AI APIs, log aggregation services.
|
|
|
|
Exemptions (result: `n/a`): pure in-process libraries (no network calls), self-hostable software where the operator controls the endpoint, and build-time-only tools.
|
|
|
|
For non-exempt libraries: verify the vendor offers an EU data region AND that the integration in `target` is configured to use it.
|
|
|
|
Result values: `ok` (vendor offers EU region, integration configured), `n/a` (no data transmission), `self-hostable` (operator-controlled endpoint), `fail` → auto-reject; short-circuit `named-consumer`.
|
|
|
|
On `fail` → set `named-consumer`, `socketRisk` to `skip` → write trace.
|
|
|
|
### Filter 8: named-consumer
|
|
|
|
Answer: **Who calls this code path today, or who is blocked waiting for it?**
|
|
|
|
A named consumer is a concrete call site that exists now or a feature blocked on this capability today. "We might want this later", "external clients could use this", and "it would be nice to have" are not named consumers.
|
|
|
|
If the only possible callers are hypothetical or future → `fail` → set `socketRisk` to `skip` → auto-reject.
|
|
|
|
Result value: `pass` or `fail`.
|
|
|
|
### Filter 9: supply-chain behavior (Socket)
|
|
|
|
**Expensive — network call. Run last in Phase 2. Short-circuit: if any earlier Phase 2 filter already rejected the library, set `socketRisk` to `skip` and proceed to the [Trace write step](#trace-write-step).**
|
|
|
|
Verify the package's supply-chain health via `socket-cli`:
|
|
|
|
```
|
|
npx socket-cli@latest scan . --json 2>&1
|
|
```
|
|
|
|
This scans the current directory's lockfile for packages installed from the target under evaluation. For a targeted single-package check before installing:
|
|
|
|
```
|
|
npx socket-cli@latest info <pkg>@<version> --json 2>&1
|
|
```
|
|
|
|
The JSON output contains an array of findings, each with a `severity` field. Cross-reference with the repo-root `.socket.json` `issueRules` to determine the classification:
|
|
|
|
| Finding severity | `.socket.json` rule | `socketRisk` value |
|
|
| ----------------------------------- | ------------------- | ------------------- |
|
|
| No findings, or only `medium`/`low` | `ignore` | `clean` |
|
|
| `high`-severity finding present | `warn` | `flagged` |
|
|
| `critical`-severity finding present | `error` | `<finding-summary>` |
|
|
|
|
Where `<finding-summary>` is a concise label for the critical finding (e.g. `"new-author-on-publish"`, `"install-scripts-added"`, `"exfiltrates-env"`).
|
|
|
|
Set `filter-results.socketRisk` in the trace frontmatter to one of these three values.
|
|
|
|
Result values:
|
|
|
|
- `clean` — no meaningful supply-chain signals; proceed to Phase 3 prompts.
|
|
- `flagged` — `high`-severity finding; document the specific signal in the trace body and decide whether to accept with justification. Not an auto-reject.
|
|
- `<finding-summary>` — `critical`-severity finding; auto-reject. This is the last filter — no further filters to skip.
|
|
|
|
---
|
|
|
|
## Skip sentinel
|
|
|
|
When a filter is short-circuited (not evaluated), write `skip` for its frontmatter value. The Zod schema validates approved traces end-to-end; rejected/partial traces may carry `skip` in fields that would normally require an enum value. The pre-commit check only validates that approved traces exist for new deps — partial traces are informational records.
|
|
|
|
## Three discussion prompts
|
|
|
|
Answer all three in the trace, regardless of filter outcome. These are not auto-reject filters; any answer is acceptable with justification.
|
|
|
|
### Prompt: replaces
|
|
|
|
What existing library or approach does this replace? New-and-old running in parallel is a smell — name the thing being retired and the retirement plan, or explain why parallel adoption is intentional and time-bounded.
|
|
|
|
### Prompt: migration-cost-out
|
|
|
|
What does ripping this back out look like 18 months from now? Rate: **mechanical** (swap one package, update call sites), **hard** (scattered integration points, data-format dependencies), or **impossible** (vendor lock-in, protocol coupling). Higher cost raises the bar for adoption.
|
|
|
|
### Prompt: alternatives-considered
|
|
|
|
Name at least two alternatives evaluated before choosing this library. For `core`-tier adoptions, this section is also duplicated into the companion ADR. If no alternatives exist, explain why (e.g., the library is the de-facto standard with no viable substitutes).
|
|
|
|
---
|
|
|
|
## Trace write step
|
|
|
|
Write the trace **unconditionally** at evaluation end — even for rejections, even for partial traces.
|
|
|
|
**Path:** `docs/library-decisions/<YYYY-MM-DD>-<package-name>.md`
|
|
|
|
Use today's date. Use the `TRACE-TEMPLATE.md` in this directory as the structural guide.
|
|
|
|
Frontmatter rules:
|
|
|
|
- `decision: approved` only if all eight filters passed. Otherwise `decision: rejected`.
|
|
- `adr: null` for feature-tier. For core-tier approvals, coordinate the ADR slug before writing (`adr: adr-NNN`).
|
|
- `verification-commands` — include the literal commands run for each filter, one per line.
|
|
- `accepted-cves: []` (empty unless you accepted a specific advisory).
|
|
- For skipped expensive filters, write `skip` for the frontmatter value and omit the prose section body or note "Not evaluated — skipped due to earlier rejection."
|
|
|
|
After writing the trace:
|
|
|
|
- For approved traces: confirm the trace is staged in the same commit as the `package.json` change. The pre-commit hook validates this.
|
|
- For rejected traces: stage the trace file alone. Do not run `pnpm add <pkg>`.
|
|
|
|
</runbook>
|
|
|
|
<output-format>
|
|
|
|
After completing the evaluation, emit a one-paragraph summary:
|
|
|
|
```
|
|
/evaluate-library result: <approved|rejected> — <package>@<version> (<tier>)
|
|
Rejection filters (if any): <filter names>
|
|
Trace written to: docs/library-decisions/<date>-<package>.md
|
|
```
|
|
|
|
</output-format>
|