docs(guides): runbook section on using Sandcastle for agent dispatch
Adds a "Using Sandcastle for agent dispatch" section between the gate table and Troubleshooting. Covers when to use / not use sandcastle, prerequisites (Docker + agent API key + .sandcastle/ config), the dispatch flow, a worked end-to-end example (plan → execute → review → manual state mutation), troubleshooting (env vars, Docker, timeouts, rejection modes), and a cost-aware planning-only variant.
This commit is contained in:
@@ -220,6 +220,183 @@ For the full design see `docs/architecture/agent-first-workflow-and-conformance.
|
||||
|
||||
---
|
||||
|
||||
## Using Sandcastle for agent dispatch
|
||||
|
||||
[Sandcastle](https://github.com/mattpocock/sandcastle) is the substrate that takes a markdown task description, hands it to a Claude / Codex agent running inside an isolated Docker sandbox, captures the agent's commits, and returns them so the orchestrator can route the diff to a reviewer agent. The repo's `pnpm work dispatch` wraps sandcastle for the manifest-first workflow.
|
||||
|
||||
### When to use Sandcastle
|
||||
|
||||
- **Routine, well-specified tasks** — adding a behaviour slice to an existing use case, migrating a feature to a new convention, scaffolding new packages. The task description is the contract; sandcastle automates the rest.
|
||||
- **Parallel work** — dispatch multiple independent tasks at once; each runs in its own sandbox branch.
|
||||
- **Reviewer-loop verification** — the reviewer agent reads the diff against the task spec and either approves or sends feedback for another implementer pass.
|
||||
|
||||
### When NOT to use Sandcastle
|
||||
|
||||
- **Exploratory / design work** — when the right answer isn't known, write it yourself. Sandcastle thrives when the task is "implement this", not "figure out what to do".
|
||||
- **Cross-cutting refactors** — dispatch is per-task; many tasks that touch unrelated files at once is better done in one human-driven session.
|
||||
- **First-time integrations** (e.g., adopting a new SDK) — better to walk through it manually, then capture the pattern as a generator for future sandcastle dispatches.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **Docker running** — sandcastle uses Docker for the sandbox by default. `docker info` should succeed.
|
||||
2. **Agent API key** — set ONE of:
|
||||
- `ANTHROPIC_API_KEY` (recommended; sandcastle's default agent is `claudeCode`)
|
||||
- `OPENAI_API_KEY` (alternative)
|
||||
3. **GitHub token** (optional) — `GITHUB_TOKEN` if you want the orchestrator to create PRs.
|
||||
4. **`.sandcastle/` config present** — already in tree:
|
||||
- `Dockerfile` — node:22 + pnpm sandbox image
|
||||
- `prd-eliciter.prompt.md` — interviews humans to draft PRDs
|
||||
- `adr-eliciter.prompt.md` — same shape, for infrastructure decisions
|
||||
- `decomposer.prompt.md` — PRD → epic + stories with generator-first task lists
|
||||
- `implementer.prompt.md` — executes one task; runs all 5 gates before committing
|
||||
- `reviewer.prompt.md` — reviews implementer's diff against AC + scope
|
||||
|
||||
### The dispatch flow
|
||||
|
||||
```
|
||||
pnpm work next → identifies the next ready story (DAG-aware)
|
||||
pnpm work dispatch → prints what WOULD be dispatched (no Sandcastle call)
|
||||
pnpm work dispatch --execute
|
||||
→ invokes sandcastle.run(implementer prompt + task spec)
|
||||
→ sandcastle returns { branch, commits, stdout, ... }
|
||||
→ orchestrator computes `git diff main..<branch>`
|
||||
→ invokes sandcastle.run(reviewer prompt + diff)
|
||||
→ reviewer returns approve / reject + notes
|
||||
→ orchestrator prints suggested state mutation
|
||||
(in v1: human ticks the bullet + commits manually)
|
||||
```
|
||||
|
||||
### Worked example — dispatch a real task
|
||||
|
||||
Suppose `pnpm work next` reports:
|
||||
|
||||
```
|
||||
auth-v1 / 02-sign-up — Sign up with email and password
|
||||
status: in-progress, tasks: 3/7
|
||||
```
|
||||
|
||||
The story file `docs/work/auth-v1/02-sign-up/_story.md` has a Tasks list with the next unchecked bullet:
|
||||
|
||||
```
|
||||
- [ ] Hash password using injected IPasswordHasher before persisting
|
||||
```
|
||||
|
||||
**Step 1 — Plan**
|
||||
|
||||
```bash
|
||||
pnpm work dispatch
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
=== Dispatch plan ===
|
||||
Epic: auth-v1
|
||||
Story: 02-sign-up — Sign up with email and password
|
||||
Bullet: - [ ] Hash password using injected IPasswordHasher before persisting
|
||||
Prompt: .sandcastle/implementer.prompt.md
|
||||
|
||||
To execute this dispatch, run:
|
||||
ANTHROPIC_API_KEY=... pnpm work dispatch --execute
|
||||
```
|
||||
|
||||
This is safe to run anywhere — it never invokes Sandcastle.
|
||||
|
||||
**Step 2 — Execute**
|
||||
|
||||
```bash
|
||||
ANTHROPIC_API_KEY=sk-ant-... pnpm work dispatch --execute
|
||||
```
|
||||
|
||||
The orchestrator:
|
||||
|
||||
1. Builds the task spec (story metadata + the current bullet + full story context)
|
||||
2. Calls `sandcastle.run({ promptFile: ".sandcastle/implementer.prompt.md", promptArgs: { TASK_FILE_CONTENT: spec }, ... })`
|
||||
3. Sandcastle pulls the Docker image, mounts the repo into `/workspace`, runs `claudeCode` with the implementer prompt template populated
|
||||
4. The implementer agent (inside the sandbox):
|
||||
- Reads the task spec
|
||||
- Runs `pnpm install --frozen-lockfile`
|
||||
- Locates the use case: `packages/auth/src/application/use-cases/sign-up.use-case.ts`
|
||||
- Writes a red test asserting `hasher.hash` is called before `repo.create`
|
||||
- Runs `pnpm test --filter @repo/auth` — sees the red test fail
|
||||
- Adds `IPasswordHasher` to the factory deps; calls `hasher.hash(input.password)` before `repo.create`
|
||||
- Runs `pnpm test --filter @repo/auth` — green
|
||||
- Runs `pnpm typecheck`, `pnpm lint`, `pnpm conformance`, `pnpm fallow:audit` — all five gates green
|
||||
- Commits on a sandbox branch (`task/02-sign-up-hash-password` or similar)
|
||||
5. Sandcastle returns: `{ branch: "task/02-sign-up-hash-password", commits: [{sha: "..."}], stdout: "...", ... }`
|
||||
|
||||
**Step 3 — Review**
|
||||
|
||||
The orchestrator immediately runs the reviewer:
|
||||
|
||||
1. Computes `git diff main..task/02-sign-up-hash-password`
|
||||
2. Calls `sandcastle.run({ promptFile: ".sandcastle/reviewer.prompt.md", promptArgs: { TASK_FILE_CONTENT: spec, DIFF: diff }, ... })`
|
||||
3. The reviewer agent reads the diff + task + story; verifies:
|
||||
- The AC bullet is satisfied (test was added; impl calls `hasher.hash`)
|
||||
- Nothing in the "Out of scope" section was touched (no drive-by edits)
|
||||
- All gates were run
|
||||
- The implementer ran `pnpm fallow:audit`
|
||||
- Generator-first was respected (no hand-rolled scaffolding)
|
||||
4. Returns `{ decision: "approve", ac_verified: [4], scope_violations: [], notes: "..." }`
|
||||
|
||||
**Step 4 — State mutation (v1: manual)**
|
||||
|
||||
The orchestrator prints:
|
||||
|
||||
```
|
||||
=== Suggested state mutation ===
|
||||
Edit docs/work/auth-v1/02-sign-up/_story.md — tick the bullet:
|
||||
- [x] Hash password using injected IPasswordHasher before persisting
|
||||
Then: pnpm work rebuild-state && git add -A && git commit -m "..."
|
||||
|
||||
(Automatic state mutation by the orchestrator is v2.)
|
||||
```
|
||||
|
||||
You (the human) then:
|
||||
|
||||
1. Merge the sandbox branch: `git merge --no-ff task/02-sign-up-hash-password`
|
||||
2. Tick the bullet in the story markdown
|
||||
3. The pre-commit hook auto-runs `pnpm work rebuild-state` + re-stages `_state.json`
|
||||
4. Push. CI runs the full gate stack (typecheck + test + lint + conformance + fallow + boundaries + visual regression).
|
||||
|
||||
### Troubleshooting Sandcastle
|
||||
|
||||
**`✗ --execute requires ANTHROPIC_API_KEY or OPENAI_API_KEY in env.`**
|
||||
— Set one. The default agent is Claude.
|
||||
|
||||
**`Error: Cannot find module '@ai-hero/sandcastle'`**
|
||||
— Run `pnpm install`. Sandcastle is a dev dependency at the workspace root.
|
||||
|
||||
**`Error: docker: command not found`** or sandcastle hangs at "starting sandbox"
|
||||
— Docker isn't running. `docker info` to confirm. On macOS, start Docker Desktop.
|
||||
|
||||
**The implementer agent times out**
|
||||
— Default `idleTimeoutSeconds` is 600 (10 minutes). For complex tasks, increase via `dispatch.mjs` (look for the `run({...})` call and add `idleTimeoutSeconds: 1800`).
|
||||
|
||||
**The reviewer rejects with `generator_skipped: true`**
|
||||
— The implementer hand-rolled what should have been generator output. Either re-dispatch (it gets the reviewer notes), or delete the implementer's diff and run `pnpm turbo gen <kind>` manually first, then dispatch the customisation as a separate task.
|
||||
|
||||
**The reviewer rejects with `scope_violations: [...]`**
|
||||
— The implementer touched files outside the AC. Re-dispatch with stricter scope; the rejection notes are passed back as context.
|
||||
|
||||
**Cost control** — each dispatch typically uses 50K–200K agent tokens depending on task complexity. The orchestrator does NOT cap retries; if you want to limit, set `max-attempts: 1` in the task's frontmatter (the orchestrator respects this in v2 — for now, just don't re-run dispatch after a reject).
|
||||
|
||||
### Cost-aware variant: planning-only loop
|
||||
|
||||
If you want sandcastle's structure without the agent spend, use planning mode + manual execution:
|
||||
|
||||
```bash
|
||||
pnpm work dispatch # prints the plan
|
||||
# (you implement the bullet manually in your editor)
|
||||
# tick the bullet in docs/work/.../...story.md
|
||||
# commit; pre-commit auto-rebuilds _state.json
|
||||
pnpm work dispatch # prints the NEXT plan
|
||||
```
|
||||
|
||||
This gives you the same DAG-aware "what's next?" without invoking any agent. Useful for exploratory work or low-budget contexts.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**`pnpm dev` refuses to boot with `ConformanceError`**
|
||||
|
||||
Reference in New Issue
Block a user