Template

Files

Danijel Martinek ecb8dd65f4 docs(guide): audit-and-compliance how-to guide

2026-05-11 16:38:23 +02:00

16 KiB

Raw Blame History

Audit logging & DPA compliance

Prerequisite: This guide assumes @repo/core-audit is scaffolded. If your project started from the slim template, run pnpm turbo gen core-package audit first, then follow the manual wiring steps in §4 below.

What DPA requires

A Data Processing Agreement (DPA) typically mandates that any system handling personal data must keep a tamper-evident record of every access to that data. The six action types covered by this template are: VIEW, CREATE, UPDATE, DELETE, EXPORT, and PERMISSION_CHANGE. Each entry must capture four required fields:

DPA field	Mapped to
Who performed the action	`actorId`, `actorType`, `actorRoles`
What was acted on	`action`, `resource.type`, `resource.id`
When it happened	`at` (server timestamp, ISO 8601)
From where the request came	`from.ipTruncated`, `from.userAgent`

What NOT to log — the DPA "exclusion list" is enforced by the AuditEntry type itself: there are no payload, body, oldValue, or newValue fields. UPDATE actions capture only changedFields (the names of modified fields, not their values). This is intentional — storing "what changed" rather than "what it changed to" prevents the audit log from becoming a secondary store of regulated data.

Retention requirements vary by jurisdiction, but a common baseline is 90 days in hot storage (queryable via Payload admin or API) and 12 months in cold archive (shipped to a log aggregator like Grafana Loki or Elasticsearch). The stdout JSON sink + log shipper pattern satisfies this: Payload holds the hot copy; the aggregator holds the archive.

Immutability is enforced by the Payload collection's update: () => false access rule. No user — including admins — can modify a written entry through the Payload API. The only write path is IAuditLog.record(). Erasure on GDPR request uses a privileged overrideAccess: true path that pseudonymizes or deletes the actorId field rather than altering the event itself.

The two-pattern model

Two complementary ways to capture a read event:

Pattern 1 — Use-case-level `record()` calls

In your feature's READ use cases, the developer explicitly calls ctx.auditLog?.record({ action: "VIEW", ... }). This gives you full control over the WHY (the reason field) and works in every context — HTTP requests, background jobs, CLI scripts, and tests.

// packages/blog/src/application/use-cases/get-article.use-case.ts
export function getArticleUseCase(
  deps: { articlesRepo: IArticlesRepository; auditLog?: AuditLogProtocol },
) {
  return async (input: GetArticleInput): Promise<GetArticleOutput> => {
    const article = await deps.articlesRepo.findById(input.id);
    await deps.auditLog?.record({
      actorId: input.userId,
      actorType: "user",
      actorRoles: input.userRoles,
      action: "VIEW",
      resource: { type: "articles", id: input.id },
      at: new Date(),
      scope: { feature: "blog", environment: process.env.NODE_ENV ?? "development", tenant: input.tenant ?? "default" },
      from: { ipTruncated: input.ipTruncated, userAgent: input.userAgent },
      containsPii: false,
      outcome: "success",
      reason: "article-page-render",
    });
    return getArticleOutputSchema.parse(article);
  };
}

Pattern 2 — Payload `afterRead` hook (automatic, opt-in)

Install createAuditAfterReadHook(...) on a collection's afterRead hook list. This captures every read of the collection — including admin UI reads, direct programmatic reads, and REST API reads — automatically, without per-use-case instrumentation.

// packages/blog/src/integrations/cms/articles.collection.ts
import { createAuditAfterReadHook } from "@repo/core-audit/hooks";

export const articlesCollection: CollectionConfig = {
  slug: "articles",
  hooks: {
    afterRead: [
      createAuditAfterReadHook({
        auditLog: ctx.auditLog,
        feature: "blog",
        tenant: "default",
      }),
    ],
  },
  // ...
};

The hook fires asynchronously (fire-and-forget) so it never blocks the read response. It uses the sentinel IP/UA "system" / "payload-admin" for admin UI reads where no request context is available.

When to use which pattern

Read source	Recommended pattern
tRPC procedure (app-facing read)	Use-case-level `record()` call — you have full request context
Payload admin UI	Hook automatically captures (no request context needed)
Background job / cron	Use-case-level `record()` call with `actorId: "system"`, sentinel IP/UA
Direct programmatic / CMS REST	Hook automatically captures
CLI / seed script	Use-case-level `record()` call with `actorId: "service-{name}"`

Use both for collections under DPA scope. The hook covers reads you might forget at the use-case layer; the use-case calls add contextual reason and accurate IP/UA.

Wiring core-audit into your app (7 steps)

After running pnpm turbo gen core-package audit, the package exists in packages/core-audit/ but is not yet wired into your app. Complete these seven steps:

Step 1 — Set `AUDIT_PSEUDONYM_SALT`

Generate a cryptographically random salt and store it in your deployment secrets manager. The salt is used for sha256 pseudonymization on GDPR erasure requests.

export AUDIT_PSEUDONYM_SALT="$(openssl rand -hex 32)"

Add to your .env (development) and to your production secrets vault. The bindAudit() function throws at startup if NODE_ENV=production and this variable is not set — intentional fail-fast behavior.

Step 2 — Mount the Payload collection

In packages/core-cms/src/payload.config.ts, import and register the append-only auditLogsCollection:

import { auditLogsCollection } from "@repo/core-audit/collection";

export default buildConfig({
  collections: [
    // ... existing collections ...
    auditLogsCollection,
  ],
});

The collection enforces update: () => false and delete: () => false at the access-control layer. The only write path is via the IAuditLog.record() API.

Step 3 — Mount the admin tRPC router

In packages/core-api/src/root.ts, import createAuditRouter and wire it to the app router. The router requires an IAuditLog instance — pass the one returned by bindAudit():

import { createAuditRouter } from "@repo/core-audit/api";

// In your router factory (called after bindAudit):
export function createAppRouter(auditLog: IAuditLog) {
  return t.router({
    // ... existing routers ...
    audit: createAuditRouter(auditLog),
  });
}

This exposes audit.eraseSubject as an admin-only tRPC mutation. Protect it with your admin auth middleware (the auditProcedure base already requires a caller-supplied auth guard — see src/integrations/api/procedures.ts).

Step 4 — Bind audit in `bind-production.ts`

In apps/web-next/src/server/bind-production.ts, call bindAudit() before any feature binder that uses ctx.auditLog:

import { bindAudit } from "@repo/core-audit/di";

// Inside your resolveProductionContext():
const { auditLog } = bindAudit(sharedContainer, {
  payloadConfig: resolvedConfig,
  sinks: ["payload", "stdout"],
});

const ctx: BindProductionContext = {
  tracer,
  logger,
  config: resolvedConfig,
  bus,
  queue,
  realtime,
  realtimeRegistry,
  auditLog,   // <- new
};

The returned auditLog is already wrapped in TraceIdEnrichingAuditLog, so every entry receives correlationId from the active OTel span automatically.

Step 5 — Install user-collection hooks (DPA recommended)

To automatically pseudonymize or delete a user's audit history when their account is deleted, install the erasure hook on your users Payload collection. In packages/auth/src/di/bind-production.ts:

if (ctx.auditLog) {
  const { createAuditErasureHook, createAuditAfterReadHook } =
    await import("@repo/core-audit/hooks");

  // Automatically erase audit entries when a user is deleted:
  usersCollection.hooks ??= {};
  usersCollection.hooks.afterDelete ??= [];
  usersCollection.hooks.afterDelete.push(
    createAuditErasureHook({ auditLog: ctx.auditLog, mode: "pseudonymize" }),
  );

  // Optionally capture VIEW events for user profile reads:
  usersCollection.hooks.afterRead ??= [];
  usersCollection.hooks.afterRead.push(
    createAuditAfterReadHook({ auditLog: ctx.auditLog, feature: "auth", tenant: "default" }),
  );
}

This step is gated on ctx.auditLog being present, so it's safely skipped in slim-template projects where @repo/core-audit has not been scaffolded.

Step 6 — Set up a log shipper

The StdoutJsonAuditLog sink writes one JSON line per audit entry to process stdout. A log shipper (Vector or Fluent Bit) reads this stdout stream and forwards entries to your centralized aggregator (Grafana Loki, Elasticsearch, Splunk, etc.). See §5 for sample configs.

Step 7 — Verify

pnpm install
pnpm lint && pnpm typecheck && pnpm test
pnpm turbo boundaries

Run your app in development mode and trigger a VIEW event. You should see an _type: "audit" JSON line in stdout within 100 ms of the action.

Sample log-shipper configs

Vector (recommended)

Vector reads the container's stdout stream, filters for audit entries (distinguished by _type: "audit"), and ships them to Grafana Loki in the EU region.

# vector.toml
[sources.app_stdout]
type = "stdin"

[transforms.parse_audit]
type = "remap"
inputs = ["app_stdout"]
source = '''
  . = parse_json!(.message)
  if ._type != "audit" { abort }
'''

[transforms.enrich_labels]
type = "remap"
inputs = ["parse_audit"]
source = '''
  .labels = { "env": .scope.environment, "feature": .scope.feature, "tenant": .scope.tenant }
'''

[sinks.loki_eu]
type = "loki"
inputs = ["enrich_labels"]
endpoint = "https://logs-prod-eu-west-0.grafana.net"
auth.strategy = "basic"
auth.user = "${LOKI_USER}"
auth.password = "${LOKI_API_KEY}"
labels.job = "audit"
labels.env = "{{ .labels.env }}"
encoding.codec = "json"

Set LOKI_USER and LOKI_API_KEY in your deployment environment. The filter if ._type != "audit" { abort } ensures only audit entries are forwarded; other stdout lines (application logs, framework output) pass through unshipped.

Fluent Bit

# fluent-bit.conf
[INPUT]
    Name              tail
    Path              /var/log/app/stdout.log
    Parser            json
    Tag               app.stdout

[FILTER]
    Name              grep
    Match             app.stdout
    Regex             _type audit

[OUTPUT]
    Name              loki
    Match             app.stdout
    Host              logs-prod-eu-west-0.grafana.net
    Port              443
    TLS               On
    Labels            job=audit,env=${AUDIT_ENV}
    HTTP_User         ${LOKI_USER}
    HTTP_Passwd       ${LOKI_API_KEY}
    line_format       json

For containerized deployments (Docker / Kubernetes), configure the log driver to write stdout to a file or use Fluent Bit's docker input plugin instead of tail.

GDPR Article 17 ("right to erasure") requires that a data subject can request deletion of their personal data. @repo/core-audit satisfies this via IAuditLog.eraseSubject(actorId, mode).

Trigger via admin tRPC

# Replace <TOKEN> with a valid admin session token and <ACTOR_ID> with the user ID
curl -X POST https://your-app.com/api/trpc/audit.eraseSubject \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <TOKEN>" \
  -d '{"json":{"actorId":"user_123","mode":"pseudonymize"}}'

The pseudonymize mode replaces actorId in every matching Payload audit entry with erased-{sha256(salt+actorId)[0:16]}. The entry itself is preserved (the event happened; only the identity is pseudonymized). The delete mode hard-deletes every entry for that actor — use only when the DPA or a court order requires it.

Trigger via the `afterDelete` hook

If you installed the erasure hook in Step 5, deleting a user via Payload admin automatically triggers pseudonymization. No manual API call is needed for the standard deletion flow.

What `eraseSubject` does NOT cover

StdoutJsonAuditLog.eraseSubject() writes a tombstone entry to stdout but cannot retroactively alter past stdout lines that have already been shipped to your aggregator. Handle this by issuing a deletion request to Loki/Elasticsearch for that actorId label after the Payload pseudonymization completes:

# Grafana Loki: delete by label selector (requires delete permission)
curl -X POST "https://logs-prod-eu-west-0.grafana.net/loki/api/v1/delete?query={job=\"audit\"}&start=0&end=$(date +%s)000000000" \
  -H "X-Scope-OrgID: ${LOKI_TENANT}" \
  --data-urlencode 'query={actorId="user_123"}'

Consult your aggregator's deletion API for the exact syntax.

Sample-week audit verification

"Can you tell who accessed any given article last Tuesday?"

Open Payload admin → Collections → Audit Logs
Filter: resource.type = "articles" and at between Monday 00:00 and Sunday 23:59
Each entry shows actorId, actorType, actorRoles, action, at, and from.ipTruncated
Cross-reference actorId with the Users collection to get the display name (keep this lookup out of the audit log itself — names are PII)

For bulk queries, use your aggregator's log search. In Grafana Loki:

{job="audit"} | json | resource_type="articles" | action="VIEW"
  | line_format "{{.at}} {{.actorId}} viewed {{.resource_id}}"

The correlationId field (populated by TraceIdEnrichingAuditLog) links each audit entry to its OTel trace, enabling pivot from the compliance timeline to the full distributed trace in Grafana Tempo or Jaeger.

Hostile-actor immutability test

The append-only guarantee is only as good as its enforcement. Verify it holds:

# 1. Record a test entry
curl -X POST http://localhost:3001/api/audit-logs \
  -H "Content-Type: application/json" \
  -d '{"actorId":"attacker","action":"VIEW","resource":{"type":"test"}}'
# Expected: 403 Forbidden — the collection's create access is API-only via IAuditLog

# 2. Try to update an existing entry via Payload REST
ENTRY_ID=$(curl -s "http://localhost:3001/api/audit-logs?limit=1" | jq -r '.docs[0].id')
curl -X PATCH "http://localhost:3001/api/audit-logs/${ENTRY_ID}" \
  -H "Content-Type: application/json" \
  -d '{"actorId":"tampered"}'
# Expected: 403 Forbidden — update: () => false

# 3. Try to delete via Payload REST
curl -X DELETE "http://localhost:3001/api/audit-logs/${ENTRY_ID}"
# Expected: 403 Forbidden — delete: () => false

Also verify that the stdout shipper is configured with an independent retention policy that does NOT depend on Payload. If a hostile actor gains DB access and truncates the audit_logs table, the shipped log lines in Loki/Elasticsearch remain as the authoritative record.

Common mistakes

Forgetting scope.tenant — the field is required, not optional. Single-tenant projects must explicitly pass tenant: "default". TypeScript will catch this at compile time if you omit it.

Setting containsPii: false on a collection that has PII — for example, a "users" profile view logs resource.type: "users" but containsPii: false. Even though the audit entry itself doesn't store PII values, the resource being accessed is PII-bearing. Set containsPii: true and list relevant categories in piiCategories: ["profile", "email"] so downstream retention systems apply the correct access policy to the audit entries themselves.

Trying to add oldValue/newValue fields — these fields do not exist on AuditEntry by design. The type system prevents this. If you need to capture the before/after state of a field for a specific compliance requirement, build a separate audit-detail mechanism outside this package — do not extend AuditEntry.

Forgetting AUDIT_PSEUDONYM_SALT in production — bindAudit() throws at startup with a clear message. Do not try to catch this error; the intent is to refuse to start rather than silently use a weak or predictable salt.

Using eraseSubject("delete") as the default — prefer "pseudonymize" for the standard GDPR erasure path. Hard delete removes all evidence that the events occurred, which can itself create compliance problems. Pseudonymization preserves the event record while making the actor unidentifiable.

Not verifying the log shipper in staging — deploy to staging with the same Vector/Fluent Bit configuration you'll use in production. Verify that audit entries appear in your aggregator before go-live. The _type: "audit" filter is your first line of defense against shipping non-audit data to the compliance log.

16 KiB Raw Blame History