.planning/phases/09-faster-write-back/09-CONTEXT.md

@@ -0,0 +1,106 @@
+# Phase 9: Faster Write-Back - Context
+
+**Gathered:** 2026-06-12
+**Status:** Ready for planning
+
+<domain>
+## Phase Boundary
+
+Make a created/edited/deleted event reach Fastmail in ~1-2s instead of waiting up to ~15s for the next interval tick, by signalling the outbox drain on enqueue (event-driven) — with every existing outbox durability guarantee intact.
+
+**In scope:** the in-process signal mechanism, its wiring into the existing drain path, and a trailing re-drain guarantee under bursts.
+
+**Out of scope (locked by roadmap — do NOT reopen):**
+- Redis for the drain (Redis stays only for list SSE). Single-process, single Node module instance by design.
+- node-cron (silently skipped ticks in the long-lived process — `setInterval` only). See `[[node-cron-skips-in-long-running-process]]`.
+- Any change to the route handlers' optimistic-202 / no-inline-CalDAV contract.
+- Any change to outbox durability semantics (fresh-etag-before-PUT, 412 conflict flow, per-uid exactly-once, edit-as-move create-before-delete, dead-letter/backoff).
+- Multi-process / multi-replica row-claim (the `isDraining` declaration comment documents the future `UPDATE ... status='processing'` path; not this phase).
+
+</domain>
+
+<decisions>
+## Implementation Decisions
+
+### Signal mechanism (locked from ROADMAP.md — restated so the planner doesn't re-derive)
+- **D-01:** New artifact is a single zero-dependency in-process `EventEmitter` at `apps/api/src/lib/outboxTrigger.ts`. No new npm dependency.
+- **D-02:** The signal funnels through the **existing** `isDraining`-guarded `runOutboxDrain()` path via a `drainRequested` flag. The trigger must NEVER call `runOutboxDrain()` directly in a way that bypasses the `isDraining` guard or escapes the error-caught wrapper (Pitfall 5 — no double-drain).
+- **D-03:** The signal is published **after** the enqueue transaction commits — never between the two inserts of an edit-as-move (Pitfall 6). For a move, publish once after both CREATE and DELETE rows are committed, so create-before-delete ordering is never raced by the signal.
+- **D-04:** Route handlers still return optimistic 202 immediately and make no inline CalDAV call; the signal is fire-and-forget.
+
+### Burst behavior — trailing re-drain (the main decision this discussion locked)
+- **D-05:** **Guarantee the last edit in a burst.** When a signal arrives while a drain is already in flight, set `drainRequested` and re-run the drain exactly once after the current one finishes (drain-again-if-requested loop), so the LAST edit in a rapid burst still lands in ~1-2s rather than waiting for the next 15s tick.
+- **D-06:** No debounce/coalesce window and no rejection of mid-drain signals. Rationale: two-user household — burst volume is tiny; correctness/snappiness of the trailing edit outweighs Fastmail request-rate politeness. (A debounce knob was considered and rejected as premature.)
+- **D-07:** Implementation must preserve exactly-once per uid when the signal and the 15s fallback overlap — the trailing re-drain reuses the same `pending AND next_attempt_at <= NOW()` selection and `isDraining` guard, so no duplicate CalDAV PUT for the same row (Success Criterion 4).
+
+### Fallback interval
+- **D-08:** **Keep the 15s `setInterval` fallback exactly as-is.** It still runs for startup catch-up and transient-error recovery (Success Criterion 5). Do not lengthen it — idle DB polling cost is negligible for this deployment and the criterion names 15s explicitly.
+
+### Verification
+- **D-09:** Prove Success Criterion 1 with an **automated integration test on the trigger wiring** (Vitest): assert that enqueuing a row publishes the signal and that `runOutboxDrain` is invoked promptly (stub/mock the CalDAV dispatch so no live Fastmail is needed). Also cover the trailing-re-drain case (signal during in-flight drain → exactly one follow-up drain) and the no-double-PUT overlap case.
+- **D-10:** No operator-stopwatch human checkpoint is required for this phase. Rationale: the dev bypass user (id 1) has no CalDAV credential/calendars, so live event-create 422s in the dev stack — see `[[dev-data-user1-no-calendars]]`; an automated wiring test is the durable, CI-guarded evidence. (Real wall-clock-to-Fastmail latency is bounded by the CalDAV round-trip regardless and is implicitly exercised in production use.)
+
+### Claude's Discretion
+- Exact shape of the `drainRequested`/re-drain loop (where the flag lives, whether the loop sits inside `runOutboxDrain`'s `finally` or in the scheduler wrapper) — planner/researcher decides, provided D-02/D-05/D-07 hold.
+- Whether the EventEmitter is a module singleton vs. a tiny custom signal object — either is fine; zero-dep is the only hard constraint.
+- Test file placement follows the existing convention (API tests live under `apps/api/tests/`, never `src/`) — see `[[api-integration-test-db]]`.
+
+</decisions>
+
+<canonical_refs>
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Phase scope & guarantees
+- `.planning/ROADMAP.md` §"Phase 9: Faster Write-Back" — goal, the 5 Success Criteria, and the 3 owned pitfalls (no double-drain, create-before-delete under concurrent enqueues, hard constraints).
+
+### Existing implementation to extend (read before changing)
+- `apps/api/src/broker/outboxWorker.ts` — `runOutboxDrain()` (the guarded drain), `isDraining` flag + its multi-process caveat comment, `startOutboxWorker()` (the 15s `setInterval`), edit-as-move ordering + durable CR-04 sibling-status gate, 412/hardfail/backoff/dead-letter handling. This is the path the signal must funnel into.
+- `apps/api/src/routes/events.ts` — the three enqueue sites (create ~L300, edit/edit-as-move transaction ~L389–432, delete ~L511) where the post-commit signal must be published, and the sync-status endpoint the PWA polls.
+- `apps/api/src/index.ts` §L136–141 — background-worker startup wiring (`startOutboxWorker` beside `startBrokerPoller`/`startReminderScheduler`); the trigger must be initialised consistently with this pattern, only under `isMainModule()`.
+
+No external specs/ADRs beyond ROADMAP.md — the durability decisions (D-04…D-12, CR-04/05, WR-01/04/06) are documented inline in `outboxWorker.ts` and `events.ts`.
+
+</canonical_refs>
+
+<code_context>
+## Existing Code Insights
+
+### Reusable Assets
+- `runOutboxDrain()` + `isDraining` guard (`outboxWorker.ts`): the signal reuses this verbatim — no parallel drain path. The trailing re-drain (D-05) is a loop around it, not a second drainer.
+- `startOutboxWorker()` `setInterval` pattern: the fallback stays; the new trigger sits alongside it and shares the same guarded entrypoint.
+- Existing post-write `triggerTargetedResync` + `RESYNC_TIMEOUT_MS` flow already makes `status='done'` mean "local cache reflects the write" — unaffected by this phase; the signal only changes *when* the drain starts.
+
+### Established Patterns
+- `setInterval`-only background workers (poller, reminderScheduler, outboxWorker) started exclusively under `isMainModule()` in `index.ts` — keeps timers out of the test process. The trigger follows the same gating.
+- Per-uid exactly-once via the `pending AND next_attempt_at <= NOW()` selection + `isDraining`; the signal path must not introduce a second selection that could double-dispatch.
+
+### Integration Points
+- Enqueue → signal: publish from `events.ts` after each `db.insert`/`db.transaction` commits (create, delete, and the move pair).
+- Signal → drain: `outboxTrigger` listener sets `drainRequested` / invokes the guarded drain in `outboxWorker.ts`.
+- Startup: initialise the trigger subscription in `index.ts` next to `startOutboxWorker()`.
+
+</code_context>
+
+<specifics>
+## Specific Ideas
+
+The 1-2s target is "signal on enqueue, don't wait for the interval" — not a hard real-time budget. Actual latency is bounded by the single CalDAV round-trip to Fastmail, which is unchanged; this phase only removes the up-to-15s queueing delay before that round-trip begins.
+
+</specifics>
+
+<deferred>
+## Deferred Ideas
+
+- **Multi-process / multi-replica outbox drain** — durable DB row-claim (`UPDATE calendar_outbox SET status='processing' WHERE id=? AND status='pending'`) instead of the in-memory `isDraining` guard. Documented in the `isDraining` declaration comment; out of scope while the deployment is single-process. Belongs to a future scaling phase, not v1.1.
+- **Debounce/coalesce knob for burst write-back** — considered and rejected for now (D-06). Revisit only if Fastmail request-rate ever becomes a concern with more members (see `[[project-nmember-expansion]]`).
+
+None of the above are in scope for Phase 9.
+
+</deferred>
+
+---
+
+*Phase: 9-Faster-Write-Back*
+*Context gathered: 2026-06-12*

.planning/phases/09-faster-write-back/09-DISCUSSION-LOG.md

@@ -0,0 +1,59 @@
+# Phase 9: Faster Write-Back - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-06-12
+**Phase:** 9-Faster-Write-Back
+**Areas discussed:** Burst handling, Verification, Fallback interval
+
+---
+
+## Burst handling
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Trailing re-drain (guarantee last edit) | If any signal arrives mid-drain, set `drainRequested` and re-run once the current drain finishes. Guarantees the LAST edit in a burst lands in ~1-2s. Slightly more CalDAV traffic under bursts. | ✓ |
+| Short debounce/coalesce | Collect signals for a 200-500ms window and drain once. Gentler on Fastmail; last edit lands a fraction slower; adds a tunable knob. | |
+| Fire-and-forget, rely on 15s for stragglers | Drop mid-drain signals; missed row waits for the next 15s tick. Simplest, but violates ~1-2s for the 2nd edit in a burst. | |
+
+**User's choice:** Trailing re-drain (guarantee last edit)
+**Notes:** Two-user household — burst volume is tiny, so snappiness/correctness of the trailing edit outweighs Fastmail request-rate politeness. Debounce knob rejected as premature.
+
+---
+
+## Verification
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Automated integration test on the trigger | Vitest test asserting enqueue publishes the signal and `runOutboxDrain` fires promptly (mock CalDAV dispatch). Fast, CI-able, no live Fastmail. | ✓ |
+| Both: integration test + operator stopwatch | Automated wiring test plus a one-time human checkpoint timing a real edit in the Fastmail native app. | |
+| Operator stopwatch only | Human checkpoint timing real edit-to-Fastmail latency; nothing in CI guards regressions. | |
+
+**User's choice:** Automated integration test on the trigger
+**Notes:** Dev bypass user (id 1) has no CalDAV credential/calendars, so live event-create 422s in the dev stack — an automated wiring test is the durable, CI-guarded evidence. Real wall-clock latency is bounded by the CalDAV round-trip and exercised in production use.
+
+---
+
+## Fallback interval
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Keep 15s | Leave the fallback at 15s exactly as today. Matches Success Criterion 5 wording; idle DB polling unchanged. | ✓ |
+| Lengthen it (e.g. 30-60s) | Slow the fallback since the signal handles the hot path; fewer idle DB queries but slower recovery for missed rows; changes a number the criterion names. | |
+
+**User's choice:** Keep 15s
+**Notes:** Idle DB polling cost is negligible for this single-process, two-user deployment.
+
+---
+
+## Claude's Discretion
+
+- Exact shape of the `drainRequested`/re-drain loop (flag location, loop inside `runOutboxDrain` `finally` vs. scheduler wrapper) — left to planner/researcher, provided the no-double-drain guard and exactly-once guarantees hold.
+- EventEmitter singleton vs. tiny custom signal object — either acceptable; zero-dependency is the only hard constraint.
+- Test file placement per existing convention (`apps/api/tests/`, not `src/`).
+
+## Deferred Ideas
+
+- Multi-process / multi-replica outbox drain via durable DB row-claim — out of scope while single-process; future scaling phase.
+- Debounce/coalesce knob for burst write-back — considered and rejected for now; revisit only if Fastmail request-rate becomes a concern with more members.