docs(guide): audit-and-compliance how-to guide

2026-05-11 16:38:23 +02:00
parent 2b0c54c013
commit ecb8dd65f4
1 changed files with 452 additions and 0 deletions
--- a/docs/guides/audit-and-compliance.md
+++ b/docs/guides/audit-and-compliance.md
@@ -0,0 +1,452 @@
+# Audit logging & DPA compliance
+
+> **Prerequisite:** This guide assumes `@repo/core-audit` is scaffolded. If your
+> project started from the slim template, run `pnpm turbo gen core-package audit`
+> first, then follow the manual wiring steps in §4 below.
+
+## What DPA requires
+
+A Data Processing Agreement (DPA) typically mandates that any system handling
+personal data must keep a tamper-evident record of every access to that data.
+The six action types covered by this template are: **VIEW**, **CREATE**,
+**UPDATE**, **DELETE**, **EXPORT**, and **PERMISSION_CHANGE**. Each entry must
+capture four required fields:
+
+| DPA field | Mapped to |
+|---|---|
+| **Who** performed the action | `actorId`, `actorType`, `actorRoles` |
+| **What** was acted on | `action`, `resource.type`, `resource.id` |
+| **When** it happened | `at` (server timestamp, ISO 8601) |
+| **From where** the request came | `from.ipTruncated`, `from.userAgent` |
+
+**What NOT to log** — the DPA "exclusion list" is enforced by the `AuditEntry`
+type itself: there are no `payload`, `body`, `oldValue`, or `newValue` fields.
+UPDATE actions capture only `changedFields` (the names of modified fields, not
+their values). This is intentional — storing "what changed" rather than "what
+it changed to" prevents the audit log from becoming a secondary store of
+regulated data.
+
+**Retention requirements** vary by jurisdiction, but a common baseline is 90 days
+in hot storage (queryable via Payload admin or API) and 12 months in cold archive
+(shipped to a log aggregator like Grafana Loki or Elasticsearch). The stdout JSON
+sink + log shipper pattern satisfies this: Payload holds the hot copy; the
+aggregator holds the archive.
+
+**Immutability** is enforced by the Payload collection's `update: () => false`
+access rule. No user — including admins — can modify a written entry through the
+Payload API. The only write path is `IAuditLog.record()`. Erasure on GDPR request
+uses a privileged `overrideAccess: true` path that pseudonymizes or deletes the
+`actorId` field rather than altering the event itself.
+
+## The two-pattern model
+
+Two complementary ways to capture a read event:
+
+### Pattern 1 — Use-case-level `record()` calls
+
+In your feature's READ use cases, the developer explicitly calls
+`ctx.auditLog?.record({ action: "VIEW", ... })`. This gives you full control
+over the WHY (the `reason` field) and works in every context — HTTP requests,
+background jobs, CLI scripts, and tests.
+
+```ts
+// packages/blog/src/application/use-cases/get-article.use-case.ts
+export function getArticleUseCase(
+  deps: { articlesRepo: IArticlesRepository; auditLog?: AuditLogProtocol },
+) {
+  return async (input: GetArticleInput): Promise<GetArticleOutput> => {
+    const article = await deps.articlesRepo.findById(input.id);
+    await deps.auditLog?.record({
+      actorId: input.userId,
+      actorType: "user",
+      actorRoles: input.userRoles,
+      action: "VIEW",
+      resource: { type: "articles", id: input.id },
+      at: new Date(),
+      scope: { feature: "blog", environment: process.env.NODE_ENV ?? "development", tenant: input.tenant ?? "default" },
+      from: { ipTruncated: input.ipTruncated, userAgent: input.userAgent },
+      containsPii: false,
+      outcome: "success",
+      reason: "article-page-render",
+    });
+    return getArticleOutputSchema.parse(article);
+  };
+}
+```
+
+### Pattern 2 — Payload `afterRead` hook (automatic, opt-in)
+
+Install `createAuditAfterReadHook(...)` on a collection's `afterRead` hook list.
+This captures **every** read of the collection — including admin UI reads, direct
+programmatic reads, and REST API reads — automatically, without per-use-case
+instrumentation.
+
+```ts
+// packages/blog/src/integrations/cms/articles.collection.ts
+import { createAuditAfterReadHook } from "@repo/core-audit/hooks";
+
+export const articlesCollection: CollectionConfig = {
+  slug: "articles",
+  hooks: {
+    afterRead: [
+      createAuditAfterReadHook({
+        auditLog: ctx.auditLog,
+        feature: "blog",
+        tenant: "default",
+      }),
+    ],
+  },
+  // ...
+};
+```
+
+The hook fires asynchronously (fire-and-forget) so it never blocks the read
+response. It uses the sentinel IP/UA `"system"` / `"payload-admin"` for admin UI
+reads where no request context is available.
+
+## When to use which pattern
+
+| Read source | Recommended pattern |
+|---|---|
+| tRPC procedure (app-facing read) | Use-case-level `record()` call — you have full request context |
+| Payload admin UI | Hook automatically captures (no request context needed) |
+| Background job / cron | Use-case-level `record()` call with `actorId: "system"`, sentinel IP/UA |
+| Direct programmatic / CMS REST | Hook automatically captures |
+| CLI / seed script | Use-case-level `record()` call with `actorId: "service-{name}"` |
+
+Use **both** for collections under DPA scope. The hook covers reads you might
+forget at the use-case layer; the use-case calls add contextual `reason` and
+accurate IP/UA.
+
+## Wiring core-audit into your app (7 steps)
+
+After running `pnpm turbo gen core-package audit`, the package exists in
+`packages/core-audit/` but is not yet wired into your app. Complete these seven
+steps:
+
+### Step 1 — Set `AUDIT_PSEUDONYM_SALT`
+
+Generate a cryptographically random salt and store it in your deployment secrets
+manager. The salt is used for sha256 pseudonymization on GDPR erasure requests.
+
+```bash
+export AUDIT_PSEUDONYM_SALT="$(openssl rand -hex 32)"
+```
+
+Add to your `.env` (development) and to your production secrets vault. The
+`bindAudit()` function throws at startup if `NODE_ENV=production` and this
+variable is not set — intentional fail-fast behavior.
+
+### Step 2 — Mount the Payload collection
+
+In `packages/core-cms/src/payload.config.ts`, import and register the
+append-only `auditLogsCollection`:
+
+```ts
+import { auditLogsCollection } from "@repo/core-audit/collection";
+
+export default buildConfig({
+  collections: [
+    // ... existing collections ...
+    auditLogsCollection,
+  ],
+});
+```
+
+The collection enforces `update: () => false` and `delete: () => false` at the
+access-control layer. The only write path is via the `IAuditLog.record()` API.
+
+### Step 3 — Mount the admin tRPC router
+
+In `packages/core-api/src/root.ts`, import `createAuditRouter` and wire it to
+the app router. The router requires an `IAuditLog` instance — pass the one
+returned by `bindAudit()`:
+
+```ts
+import { createAuditRouter } from "@repo/core-audit/api";
+
+// In your router factory (called after bindAudit):
+export function createAppRouter(auditLog: IAuditLog) {
+  return t.router({
+    // ... existing routers ...
+    audit: createAuditRouter(auditLog),
+  });
+}
+```
+
+This exposes `audit.eraseSubject` as an admin-only tRPC mutation. Protect it
+with your admin auth middleware (the `auditProcedure` base already requires a
+caller-supplied auth guard — see `src/integrations/api/procedures.ts`).
+
+### Step 4 — Bind audit in `bind-production.ts`
+
+In `apps/web-next/src/server/bind-production.ts`, call `bindAudit()` before
+any feature binder that uses `ctx.auditLog`:
+
+```ts
+import { bindAudit } from "@repo/core-audit/di";
+
+// Inside your resolveProductionContext():
+const { auditLog } = bindAudit(sharedContainer, {
+  payloadConfig: resolvedConfig,
+  sinks: ["payload", "stdout"],
+});
+
+const ctx: BindProductionContext = {
+  tracer,
+  logger,
+  config: resolvedConfig,
+  bus,
+  queue,
+  realtime,
+  realtimeRegistry,
+  auditLog,   // <- new
+};
+```
+
+The returned `auditLog` is already wrapped in `TraceIdEnrichingAuditLog`, so
+every entry receives `correlationId` from the active OTel span automatically.
+
+### Step 5 — Install user-collection hooks (DPA recommended)
+
+To automatically pseudonymize or delete a user's audit history when their account
+is deleted, install the erasure hook on your users Payload collection. In
+`packages/auth/src/di/bind-production.ts`:
+
+```ts
+if (ctx.auditLog) {
+  const { createAuditErasureHook, createAuditAfterReadHook } =
+    await import("@repo/core-audit/hooks");
+
+  // Automatically erase audit entries when a user is deleted:
+  usersCollection.hooks ??= {};
+  usersCollection.hooks.afterDelete ??= [];
+  usersCollection.hooks.afterDelete.push(
+    createAuditErasureHook({ auditLog: ctx.auditLog, mode: "pseudonymize" }),
+  );
+
+  // Optionally capture VIEW events for user profile reads:
+  usersCollection.hooks.afterRead ??= [];
+  usersCollection.hooks.afterRead.push(
+    createAuditAfterReadHook({ auditLog: ctx.auditLog, feature: "auth", tenant: "default" }),
+  );
+}
+```
+
+This step is gated on `ctx.auditLog` being present, so it's safely skipped in
+slim-template projects where `@repo/core-audit` has not been scaffolded.
+
+### Step 6 — Set up a log shipper
+
+The `StdoutJsonAuditLog` sink writes one JSON line per audit entry to process
+stdout. A log shipper (Vector or Fluent Bit) reads this stdout stream and
+forwards entries to your centralized aggregator (Grafana Loki, Elasticsearch,
+Splunk, etc.). See §5 for sample configs.
+
+### Step 7 — Verify
+
+```bash
+pnpm install
+pnpm lint && pnpm typecheck && pnpm test
+pnpm turbo boundaries
+```
+
+Run your app in development mode and trigger a VIEW event. You should see an
+`_type: "audit"` JSON line in stdout within 100 ms of the action.
+
+## Sample log-shipper configs
+
+### Vector (recommended)
+
+Vector reads the container's stdout stream, filters for audit entries
+(distinguished by `_type: "audit"`), and ships them to Grafana Loki in the EU
+region.
+
+```toml
+# vector.toml
+[sources.app_stdout]
+type = "stdin"
+
+[transforms.parse_audit]
+type = "remap"
+inputs = ["app_stdout"]
+source = '''
+  . = parse_json!(.message)
+  if ._type != "audit" { abort }
+'''
+
+[transforms.enrich_labels]
+type = "remap"
+inputs = ["parse_audit"]
+source = '''
+  .labels = { "env": .scope.environment, "feature": .scope.feature, "tenant": .scope.tenant }
+'''
+
+[sinks.loki_eu]
+type = "loki"
+inputs = ["enrich_labels"]
+endpoint = "https://logs-prod-eu-west-0.grafana.net"
+auth.strategy = "basic"
+auth.user = "${LOKI_USER}"
+auth.password = "${LOKI_API_KEY}"
+labels.job = "audit"
+labels.env = "{{ .labels.env }}"
+encoding.codec = "json"
+```
+
+Set `LOKI_USER` and `LOKI_API_KEY` in your deployment environment. The filter
+`if ._type != "audit" { abort }` ensures only audit entries are forwarded;
+other stdout lines (application logs, framework output) pass through unshipped.
+
+### Fluent Bit
+
+```ini
+# fluent-bit.conf
+[INPUT]
+    Name              tail
+    Path              /var/log/app/stdout.log
+    Parser            json
+    Tag               app.stdout
+
+[FILTER]
+    Name              grep
+    Match             app.stdout
+    Regex             _type audit
+
+[OUTPUT]
+    Name              loki
+    Match             app.stdout
+    Host              logs-prod-eu-west-0.grafana.net
+    Port              443
+    TLS               On
+    Labels            job=audit,env=${AUDIT_ENV}
+    HTTP_User         ${LOKI_USER}
+    HTTP_Passwd       ${LOKI_API_KEY}
+    line_format       json
+```
+
+For containerized deployments (Docker / Kubernetes), configure the log driver to
+write stdout to a file or use Fluent Bit's `docker` input plugin instead of `tail`.
+
+## GDPR erasure
+
+GDPR Article 17 ("right to erasure") requires that a data subject can request
+deletion of their personal data. `@repo/core-audit` satisfies this via
+`IAuditLog.eraseSubject(actorId, mode)`.
+
+### Trigger via admin tRPC
+
+```bash
+# Replace <TOKEN> with a valid admin session token and <ACTOR_ID> with the user ID
+curl -X POST https://your-app.com/api/trpc/audit.eraseSubject \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer <TOKEN>" \
+  -d '{"json":{"actorId":"user_123","mode":"pseudonymize"}}'
+```
+
+The `pseudonymize` mode replaces `actorId` in every matching Payload audit entry
+with `erased-{sha256(salt+actorId)[0:16]}`. The entry itself is preserved (the
+event happened; only the identity is pseudonymized). The `delete` mode hard-deletes
+every entry for that actor — use only when the DPA or a court order requires it.
+
+### Trigger via the `afterDelete` hook
+
+If you installed the erasure hook in Step 5, deleting a user via Payload admin
+automatically triggers pseudonymization. No manual API call is needed for the
+standard deletion flow.
+
+### What `eraseSubject` does NOT cover
+
+`StdoutJsonAuditLog.eraseSubject()` writes a tombstone entry to stdout but cannot
+retroactively alter past stdout lines that have already been shipped to your
+aggregator. Handle this by issuing a deletion request to Loki/Elasticsearch for
+that `actorId` label after the Payload pseudonymization completes:
+
+```bash
+# Grafana Loki: delete by label selector (requires delete permission)
+curl -X POST "https://logs-prod-eu-west-0.grafana.net/loki/api/v1/delete?query={job=\"audit\"}&start=0&end=$(date +%s)000000000" \
+  -H "X-Scope-OrgID: ${LOKI_TENANT}" \
+  --data-urlencode 'query={actorId="user_123"}'
+```
+
+Consult your aggregator's deletion API for the exact syntax.
+
+## Sample-week audit verification
+
+"Can you tell who accessed any given article last Tuesday?"
+
+1. Open Payload admin → Collections → Audit Logs
+2. Filter: `resource.type = "articles"` and `at` between Monday 00:00 and Sunday 23:59
+3. Each entry shows `actorId`, `actorType`, `actorRoles`, `action`, `at`, and `from.ipTruncated`
+4. Cross-reference `actorId` with the Users collection to get the display name (keep this lookup out of the audit log itself — names are PII)
+
+For bulk queries, use your aggregator's log search. In Grafana Loki:
+
+```logql
+{job="audit"} | json | resource_type="articles" | action="VIEW"
+  | line_format "{{.at}} {{.actorId}} viewed {{.resource_id}}"
+```
+
+The `correlationId` field (populated by `TraceIdEnrichingAuditLog`) links each
+audit entry to its OTel trace, enabling pivot from the compliance timeline to the
+full distributed trace in Grafana Tempo or Jaeger.
+
+## Hostile-actor immutability test
+
+The append-only guarantee is only as good as its enforcement. Verify it holds:
+
+```bash
+# 1. Record a test entry
+curl -X POST http://localhost:3001/api/audit-logs \
+  -H "Content-Type: application/json" \
+  -d '{"actorId":"attacker","action":"VIEW","resource":{"type":"test"}}'
+# Expected: 403 Forbidden — the collection's create access is API-only via IAuditLog
+
+# 2. Try to update an existing entry via Payload REST
+ENTRY_ID=$(curl -s "http://localhost:3001/api/audit-logs?limit=1" | jq -r '.docs[0].id')
+curl -X PATCH "http://localhost:3001/api/audit-logs/${ENTRY_ID}" \
+  -H "Content-Type: application/json" \
+  -d '{"actorId":"tampered"}'
+# Expected: 403 Forbidden — update: () => false
+
+# 3. Try to delete via Payload REST
+curl -X DELETE "http://localhost:3001/api/audit-logs/${ENTRY_ID}"
+# Expected: 403 Forbidden — delete: () => false
+```
+
+Also verify that the stdout shipper is configured with an independent retention
+policy that does NOT depend on Payload. If a hostile actor gains DB access and
+truncates the `audit_logs` table, the shipped log lines in Loki/Elasticsearch
+remain as the authoritative record.
+
+## Common mistakes
+
+**Forgetting `scope.tenant`** — the field is required, not optional. Single-tenant
+projects must explicitly pass `tenant: "default"`. TypeScript will catch this at
+compile time if you omit it.
+
+**Setting `containsPii: false` on a collection that has PII** — for example, a
+"users" profile view logs `resource.type: "users"` but `containsPii: false`.
+Even though the audit entry itself doesn't store PII values, the *resource* being
+accessed is PII-bearing. Set `containsPii: true` and list relevant categories in
+`piiCategories: ["profile", "email"]` so downstream retention systems apply the
+correct access policy to the audit entries themselves.
+
+**Trying to add `oldValue`/`newValue` fields** — these fields do not exist on
+`AuditEntry` by design. The type system prevents this. If you need to capture the
+before/after state of a field for a specific compliance requirement, build a
+separate audit-detail mechanism outside this package — do not extend `AuditEntry`.
+
+**Forgetting `AUDIT_PSEUDONYM_SALT` in production** — `bindAudit()` throws at
+startup with a clear message. Do not try to catch this error; the intent is to
+refuse to start rather than silently use a weak or predictable salt.
+
+**Using `eraseSubject("delete")` as the default** — prefer `"pseudonymize"` for
+the standard GDPR erasure path. Hard delete removes all evidence that the events
+occurred, which can itself create compliance problems. Pseudonymization preserves
+the event record while making the actor unidentifiable.
+
+**Not verifying the log shipper in staging** — deploy to staging with the same
+Vector/Fluent Bit configuration you'll use in production. Verify that audit
+entries appear in your aggregator before go-live. The `_type: "audit"` filter is
+your first line of defense against shipping non-audit data to the compliance log.