From ecb8dd65f462d42ae3960a4d3bdf430bc6d1f249 Mon Sep 17 00:00:00 2001 From: Danijel Martinek Date: Mon, 11 May 2026 16:38:23 +0200 Subject: [PATCH] docs(guide): audit-and-compliance how-to guide --- docs/guides/audit-and-compliance.md | 452 ++++++++++++++++++++++++++++ 1 file changed, 452 insertions(+) create mode 100644 docs/guides/audit-and-compliance.md diff --git a/docs/guides/audit-and-compliance.md b/docs/guides/audit-and-compliance.md new file mode 100644 index 0000000..b4f5e3b --- /dev/null +++ b/docs/guides/audit-and-compliance.md @@ -0,0 +1,452 @@ +# Audit logging & DPA compliance + +> **Prerequisite:** This guide assumes `@repo/core-audit` is scaffolded. If your +> project started from the slim template, run `pnpm turbo gen core-package audit` +> first, then follow the manual wiring steps in §4 below. + +## What DPA requires + +A Data Processing Agreement (DPA) typically mandates that any system handling +personal data must keep a tamper-evident record of every access to that data. +The six action types covered by this template are: **VIEW**, **CREATE**, +**UPDATE**, **DELETE**, **EXPORT**, and **PERMISSION_CHANGE**. Each entry must +capture four required fields: + +| DPA field | Mapped to | +|---|---| +| **Who** performed the action | `actorId`, `actorType`, `actorRoles` | +| **What** was acted on | `action`, `resource.type`, `resource.id` | +| **When** it happened | `at` (server timestamp, ISO 8601) | +| **From where** the request came | `from.ipTruncated`, `from.userAgent` | + +**What NOT to log** — the DPA "exclusion list" is enforced by the `AuditEntry` +type itself: there are no `payload`, `body`, `oldValue`, or `newValue` fields. +UPDATE actions capture only `changedFields` (the names of modified fields, not +their values). This is intentional — storing "what changed" rather than "what +it changed to" prevents the audit log from becoming a secondary store of +regulated data. + +**Retention requirements** vary by jurisdiction, but a common baseline is 90 days +in hot storage (queryable via Payload admin or API) and 12 months in cold archive +(shipped to a log aggregator like Grafana Loki or Elasticsearch). The stdout JSON +sink + log shipper pattern satisfies this: Payload holds the hot copy; the +aggregator holds the archive. + +**Immutability** is enforced by the Payload collection's `update: () => false` +access rule. No user — including admins — can modify a written entry through the +Payload API. The only write path is `IAuditLog.record()`. Erasure on GDPR request +uses a privileged `overrideAccess: true` path that pseudonymizes or deletes the +`actorId` field rather than altering the event itself. + +## The two-pattern model + +Two complementary ways to capture a read event: + +### Pattern 1 — Use-case-level `record()` calls + +In your feature's READ use cases, the developer explicitly calls +`ctx.auditLog?.record({ action: "VIEW", ... })`. This gives you full control +over the WHY (the `reason` field) and works in every context — HTTP requests, +background jobs, CLI scripts, and tests. + +```ts +// packages/blog/src/application/use-cases/get-article.use-case.ts +export function getArticleUseCase( + deps: { articlesRepo: IArticlesRepository; auditLog?: AuditLogProtocol }, +) { + return async (input: GetArticleInput): Promise => { + const article = await deps.articlesRepo.findById(input.id); + await deps.auditLog?.record({ + actorId: input.userId, + actorType: "user", + actorRoles: input.userRoles, + action: "VIEW", + resource: { type: "articles", id: input.id }, + at: new Date(), + scope: { feature: "blog", environment: process.env.NODE_ENV ?? "development", tenant: input.tenant ?? "default" }, + from: { ipTruncated: input.ipTruncated, userAgent: input.userAgent }, + containsPii: false, + outcome: "success", + reason: "article-page-render", + }); + return getArticleOutputSchema.parse(article); + }; +} +``` + +### Pattern 2 — Payload `afterRead` hook (automatic, opt-in) + +Install `createAuditAfterReadHook(...)` on a collection's `afterRead` hook list. +This captures **every** read of the collection — including admin UI reads, direct +programmatic reads, and REST API reads — automatically, without per-use-case +instrumentation. + +```ts +// packages/blog/src/integrations/cms/articles.collection.ts +import { createAuditAfterReadHook } from "@repo/core-audit/hooks"; + +export const articlesCollection: CollectionConfig = { + slug: "articles", + hooks: { + afterRead: [ + createAuditAfterReadHook({ + auditLog: ctx.auditLog, + feature: "blog", + tenant: "default", + }), + ], + }, + // ... +}; +``` + +The hook fires asynchronously (fire-and-forget) so it never blocks the read +response. It uses the sentinel IP/UA `"system"` / `"payload-admin"` for admin UI +reads where no request context is available. + +## When to use which pattern + +| Read source | Recommended pattern | +|---|---| +| tRPC procedure (app-facing read) | Use-case-level `record()` call — you have full request context | +| Payload admin UI | Hook automatically captures (no request context needed) | +| Background job / cron | Use-case-level `record()` call with `actorId: "system"`, sentinel IP/UA | +| Direct programmatic / CMS REST | Hook automatically captures | +| CLI / seed script | Use-case-level `record()` call with `actorId: "service-{name}"` | + +Use **both** for collections under DPA scope. The hook covers reads you might +forget at the use-case layer; the use-case calls add contextual `reason` and +accurate IP/UA. + +## Wiring core-audit into your app (7 steps) + +After running `pnpm turbo gen core-package audit`, the package exists in +`packages/core-audit/` but is not yet wired into your app. Complete these seven +steps: + +### Step 1 — Set `AUDIT_PSEUDONYM_SALT` + +Generate a cryptographically random salt and store it in your deployment secrets +manager. The salt is used for sha256 pseudonymization on GDPR erasure requests. + +```bash +export AUDIT_PSEUDONYM_SALT="$(openssl rand -hex 32)" +``` + +Add to your `.env` (development) and to your production secrets vault. The +`bindAudit()` function throws at startup if `NODE_ENV=production` and this +variable is not set — intentional fail-fast behavior. + +### Step 2 — Mount the Payload collection + +In `packages/core-cms/src/payload.config.ts`, import and register the +append-only `auditLogsCollection`: + +```ts +import { auditLogsCollection } from "@repo/core-audit/collection"; + +export default buildConfig({ + collections: [ + // ... existing collections ... + auditLogsCollection, + ], +}); +``` + +The collection enforces `update: () => false` and `delete: () => false` at the +access-control layer. The only write path is via the `IAuditLog.record()` API. + +### Step 3 — Mount the admin tRPC router + +In `packages/core-api/src/root.ts`, import `createAuditRouter` and wire it to +the app router. The router requires an `IAuditLog` instance — pass the one +returned by `bindAudit()`: + +```ts +import { createAuditRouter } from "@repo/core-audit/api"; + +// In your router factory (called after bindAudit): +export function createAppRouter(auditLog: IAuditLog) { + return t.router({ + // ... existing routers ... + audit: createAuditRouter(auditLog), + }); +} +``` + +This exposes `audit.eraseSubject` as an admin-only tRPC mutation. Protect it +with your admin auth middleware (the `auditProcedure` base already requires a +caller-supplied auth guard — see `src/integrations/api/procedures.ts`). + +### Step 4 — Bind audit in `bind-production.ts` + +In `apps/web-next/src/server/bind-production.ts`, call `bindAudit()` before +any feature binder that uses `ctx.auditLog`: + +```ts +import { bindAudit } from "@repo/core-audit/di"; + +// Inside your resolveProductionContext(): +const { auditLog } = bindAudit(sharedContainer, { + payloadConfig: resolvedConfig, + sinks: ["payload", "stdout"], +}); + +const ctx: BindProductionContext = { + tracer, + logger, + config: resolvedConfig, + bus, + queue, + realtime, + realtimeRegistry, + auditLog, // <- new +}; +``` + +The returned `auditLog` is already wrapped in `TraceIdEnrichingAuditLog`, so +every entry receives `correlationId` from the active OTel span automatically. + +### Step 5 — Install user-collection hooks (DPA recommended) + +To automatically pseudonymize or delete a user's audit history when their account +is deleted, install the erasure hook on your users Payload collection. In +`packages/auth/src/di/bind-production.ts`: + +```ts +if (ctx.auditLog) { + const { createAuditErasureHook, createAuditAfterReadHook } = + await import("@repo/core-audit/hooks"); + + // Automatically erase audit entries when a user is deleted: + usersCollection.hooks ??= {}; + usersCollection.hooks.afterDelete ??= []; + usersCollection.hooks.afterDelete.push( + createAuditErasureHook({ auditLog: ctx.auditLog, mode: "pseudonymize" }), + ); + + // Optionally capture VIEW events for user profile reads: + usersCollection.hooks.afterRead ??= []; + usersCollection.hooks.afterRead.push( + createAuditAfterReadHook({ auditLog: ctx.auditLog, feature: "auth", tenant: "default" }), + ); +} +``` + +This step is gated on `ctx.auditLog` being present, so it's safely skipped in +slim-template projects where `@repo/core-audit` has not been scaffolded. + +### Step 6 — Set up a log shipper + +The `StdoutJsonAuditLog` sink writes one JSON line per audit entry to process +stdout. A log shipper (Vector or Fluent Bit) reads this stdout stream and +forwards entries to your centralized aggregator (Grafana Loki, Elasticsearch, +Splunk, etc.). See §5 for sample configs. + +### Step 7 — Verify + +```bash +pnpm install +pnpm lint && pnpm typecheck && pnpm test +pnpm turbo boundaries +``` + +Run your app in development mode and trigger a VIEW event. You should see an +`_type: "audit"` JSON line in stdout within 100 ms of the action. + +## Sample log-shipper configs + +### Vector (recommended) + +Vector reads the container's stdout stream, filters for audit entries +(distinguished by `_type: "audit"`), and ships them to Grafana Loki in the EU +region. + +```toml +# vector.toml +[sources.app_stdout] +type = "stdin" + +[transforms.parse_audit] +type = "remap" +inputs = ["app_stdout"] +source = ''' + . = parse_json!(.message) + if ._type != "audit" { abort } +''' + +[transforms.enrich_labels] +type = "remap" +inputs = ["parse_audit"] +source = ''' + .labels = { "env": .scope.environment, "feature": .scope.feature, "tenant": .scope.tenant } +''' + +[sinks.loki_eu] +type = "loki" +inputs = ["enrich_labels"] +endpoint = "https://logs-prod-eu-west-0.grafana.net" +auth.strategy = "basic" +auth.user = "${LOKI_USER}" +auth.password = "${LOKI_API_KEY}" +labels.job = "audit" +labels.env = "{{ .labels.env }}" +encoding.codec = "json" +``` + +Set `LOKI_USER` and `LOKI_API_KEY` in your deployment environment. The filter +`if ._type != "audit" { abort }` ensures only audit entries are forwarded; +other stdout lines (application logs, framework output) pass through unshipped. + +### Fluent Bit + +```ini +# fluent-bit.conf +[INPUT] + Name tail + Path /var/log/app/stdout.log + Parser json + Tag app.stdout + +[FILTER] + Name grep + Match app.stdout + Regex _type audit + +[OUTPUT] + Name loki + Match app.stdout + Host logs-prod-eu-west-0.grafana.net + Port 443 + TLS On + Labels job=audit,env=${AUDIT_ENV} + HTTP_User ${LOKI_USER} + HTTP_Passwd ${LOKI_API_KEY} + line_format json +``` + +For containerized deployments (Docker / Kubernetes), configure the log driver to +write stdout to a file or use Fluent Bit's `docker` input plugin instead of `tail`. + +## GDPR erasure + +GDPR Article 17 ("right to erasure") requires that a data subject can request +deletion of their personal data. `@repo/core-audit` satisfies this via +`IAuditLog.eraseSubject(actorId, mode)`. + +### Trigger via admin tRPC + +```bash +# Replace with a valid admin session token and with the user ID +curl -X POST https://your-app.com/api/trpc/audit.eraseSubject \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{"json":{"actorId":"user_123","mode":"pseudonymize"}}' +``` + +The `pseudonymize` mode replaces `actorId` in every matching Payload audit entry +with `erased-{sha256(salt+actorId)[0:16]}`. The entry itself is preserved (the +event happened; only the identity is pseudonymized). The `delete` mode hard-deletes +every entry for that actor — use only when the DPA or a court order requires it. + +### Trigger via the `afterDelete` hook + +If you installed the erasure hook in Step 5, deleting a user via Payload admin +automatically triggers pseudonymization. No manual API call is needed for the +standard deletion flow. + +### What `eraseSubject` does NOT cover + +`StdoutJsonAuditLog.eraseSubject()` writes a tombstone entry to stdout but cannot +retroactively alter past stdout lines that have already been shipped to your +aggregator. Handle this by issuing a deletion request to Loki/Elasticsearch for +that `actorId` label after the Payload pseudonymization completes: + +```bash +# Grafana Loki: delete by label selector (requires delete permission) +curl -X POST "https://logs-prod-eu-west-0.grafana.net/loki/api/v1/delete?query={job=\"audit\"}&start=0&end=$(date +%s)000000000" \ + -H "X-Scope-OrgID: ${LOKI_TENANT}" \ + --data-urlencode 'query={actorId="user_123"}' +``` + +Consult your aggregator's deletion API for the exact syntax. + +## Sample-week audit verification + +"Can you tell who accessed any given article last Tuesday?" + +1. Open Payload admin → Collections → Audit Logs +2. Filter: `resource.type = "articles"` and `at` between Monday 00:00 and Sunday 23:59 +3. Each entry shows `actorId`, `actorType`, `actorRoles`, `action`, `at`, and `from.ipTruncated` +4. Cross-reference `actorId` with the Users collection to get the display name (keep this lookup out of the audit log itself — names are PII) + +For bulk queries, use your aggregator's log search. In Grafana Loki: + +```logql +{job="audit"} | json | resource_type="articles" | action="VIEW" + | line_format "{{.at}} {{.actorId}} viewed {{.resource_id}}" +``` + +The `correlationId` field (populated by `TraceIdEnrichingAuditLog`) links each +audit entry to its OTel trace, enabling pivot from the compliance timeline to the +full distributed trace in Grafana Tempo or Jaeger. + +## Hostile-actor immutability test + +The append-only guarantee is only as good as its enforcement. Verify it holds: + +```bash +# 1. Record a test entry +curl -X POST http://localhost:3001/api/audit-logs \ + -H "Content-Type: application/json" \ + -d '{"actorId":"attacker","action":"VIEW","resource":{"type":"test"}}' +# Expected: 403 Forbidden — the collection's create access is API-only via IAuditLog + +# 2. Try to update an existing entry via Payload REST +ENTRY_ID=$(curl -s "http://localhost:3001/api/audit-logs?limit=1" | jq -r '.docs[0].id') +curl -X PATCH "http://localhost:3001/api/audit-logs/${ENTRY_ID}" \ + -H "Content-Type: application/json" \ + -d '{"actorId":"tampered"}' +# Expected: 403 Forbidden — update: () => false + +# 3. Try to delete via Payload REST +curl -X DELETE "http://localhost:3001/api/audit-logs/${ENTRY_ID}" +# Expected: 403 Forbidden — delete: () => false +``` + +Also verify that the stdout shipper is configured with an independent retention +policy that does NOT depend on Payload. If a hostile actor gains DB access and +truncates the `audit_logs` table, the shipped log lines in Loki/Elasticsearch +remain as the authoritative record. + +## Common mistakes + +**Forgetting `scope.tenant`** — the field is required, not optional. Single-tenant +projects must explicitly pass `tenant: "default"`. TypeScript will catch this at +compile time if you omit it. + +**Setting `containsPii: false` on a collection that has PII** — for example, a +"users" profile view logs `resource.type: "users"` but `containsPii: false`. +Even though the audit entry itself doesn't store PII values, the *resource* being +accessed is PII-bearing. Set `containsPii: true` and list relevant categories in +`piiCategories: ["profile", "email"]` so downstream retention systems apply the +correct access policy to the audit entries themselves. + +**Trying to add `oldValue`/`newValue` fields** — these fields do not exist on +`AuditEntry` by design. The type system prevents this. If you need to capture the +before/after state of a field for a specific compliance requirement, build a +separate audit-detail mechanism outside this package — do not extend `AuditEntry`. + +**Forgetting `AUDIT_PSEUDONYM_SALT` in production** — `bindAudit()` throws at +startup with a clear message. Do not try to catch this error; the intent is to +refuse to start rather than silently use a weak or predictable salt. + +**Using `eraseSubject("delete")` as the default** — prefer `"pseudonymize"` for +the standard GDPR erasure path. Hard delete removes all evidence that the events +occurred, which can itself create compliance problems. Pseudonymization preserves +the event record while making the actor unidentifiable. + +**Not verifying the log shipper in staging** — deploy to staging with the same +Vector/Fluent Bit configuration you'll use in production. Verify that audit +entries appear in your aggregator before go-live. The `_type: "audit"` filter is +your first line of defense against shipping non-audit data to the compliance log.