From 2a11e86bca47082bcaebc28af77471d0b7378339 Mon Sep 17 00:00:00 2001 From: Lucas Berger Date: Fri, 12 Jun 2026 14:02:21 -0400 Subject: [PATCH] docs(09): capture phase context --- .../phases/09-faster-write-back/09-CONTEXT.md | 106 ++++++++++++++++++ .../09-faster-write-back/09-DISCUSSION-LOG.md | 59 ++++++++++ 2 files changed, 165 insertions(+) create mode 100644 .planning/phases/09-faster-write-back/09-CONTEXT.md create mode 100644 .planning/phases/09-faster-write-back/09-DISCUSSION-LOG.md diff --git a/.planning/phases/09-faster-write-back/09-CONTEXT.md b/.planning/phases/09-faster-write-back/09-CONTEXT.md new file mode 100644 index 0000000..eb544fd --- /dev/null +++ b/.planning/phases/09-faster-write-back/09-CONTEXT.md @@ -0,0 +1,106 @@ +# Phase 9: Faster Write-Back - Context + +**Gathered:** 2026-06-12 +**Status:** Ready for planning + + +## Phase Boundary + +Make a created/edited/deleted event reach Fastmail in ~1-2s instead of waiting up to ~15s for the next interval tick, by signalling the outbox drain on enqueue (event-driven) — with every existing outbox durability guarantee intact. + +**In scope:** the in-process signal mechanism, its wiring into the existing drain path, and a trailing re-drain guarantee under bursts. + +**Out of scope (locked by roadmap — do NOT reopen):** +- Redis for the drain (Redis stays only for list SSE). Single-process, single Node module instance by design. +- node-cron (silently skipped ticks in the long-lived process — `setInterval` only). See `[[node-cron-skips-in-long-running-process]]`. +- Any change to the route handlers' optimistic-202 / no-inline-CalDAV contract. +- Any change to outbox durability semantics (fresh-etag-before-PUT, 412 conflict flow, per-uid exactly-once, edit-as-move create-before-delete, dead-letter/backoff). +- Multi-process / multi-replica row-claim (the `isDraining` declaration comment documents the future `UPDATE ... status='processing'` path; not this phase). + + + + +## Implementation Decisions + +### Signal mechanism (locked from ROADMAP.md — restated so the planner doesn't re-derive) +- **D-01:** New artifact is a single zero-dependency in-process `EventEmitter` at `apps/api/src/lib/outboxTrigger.ts`. No new npm dependency. +- **D-02:** The signal funnels through the **existing** `isDraining`-guarded `runOutboxDrain()` path via a `drainRequested` flag. The trigger must NEVER call `runOutboxDrain()` directly in a way that bypasses the `isDraining` guard or escapes the error-caught wrapper (Pitfall 5 — no double-drain). +- **D-03:** The signal is published **after** the enqueue transaction commits — never between the two inserts of an edit-as-move (Pitfall 6). For a move, publish once after both CREATE and DELETE rows are committed, so create-before-delete ordering is never raced by the signal. +- **D-04:** Route handlers still return optimistic 202 immediately and make no inline CalDAV call; the signal is fire-and-forget. + +### Burst behavior — trailing re-drain (the main decision this discussion locked) +- **D-05:** **Guarantee the last edit in a burst.** When a signal arrives while a drain is already in flight, set `drainRequested` and re-run the drain exactly once after the current one finishes (drain-again-if-requested loop), so the LAST edit in a rapid burst still lands in ~1-2s rather than waiting for the next 15s tick. +- **D-06:** No debounce/coalesce window and no rejection of mid-drain signals. Rationale: two-user household — burst volume is tiny; correctness/snappiness of the trailing edit outweighs Fastmail request-rate politeness. (A debounce knob was considered and rejected as premature.) +- **D-07:** Implementation must preserve exactly-once per uid when the signal and the 15s fallback overlap — the trailing re-drain reuses the same `pending AND next_attempt_at <= NOW()` selection and `isDraining` guard, so no duplicate CalDAV PUT for the same row (Success Criterion 4). + +### Fallback interval +- **D-08:** **Keep the 15s `setInterval` fallback exactly as-is.** It still runs for startup catch-up and transient-error recovery (Success Criterion 5). Do not lengthen it — idle DB polling cost is negligible for this deployment and the criterion names 15s explicitly. + +### Verification +- **D-09:** Prove Success Criterion 1 with an **automated integration test on the trigger wiring** (Vitest): assert that enqueuing a row publishes the signal and that `runOutboxDrain` is invoked promptly (stub/mock the CalDAV dispatch so no live Fastmail is needed). Also cover the trailing-re-drain case (signal during in-flight drain → exactly one follow-up drain) and the no-double-PUT overlap case. +- **D-10:** No operator-stopwatch human checkpoint is required for this phase. Rationale: the dev bypass user (id 1) has no CalDAV credential/calendars, so live event-create 422s in the dev stack — see `[[dev-data-user1-no-calendars]]`; an automated wiring test is the durable, CI-guarded evidence. (Real wall-clock-to-Fastmail latency is bounded by the CalDAV round-trip regardless and is implicitly exercised in production use.) + +### Claude's Discretion +- Exact shape of the `drainRequested`/re-drain loop (where the flag lives, whether the loop sits inside `runOutboxDrain`'s `finally` or in the scheduler wrapper) — planner/researcher decides, provided D-02/D-05/D-07 hold. +- Whether the EventEmitter is a module singleton vs. a tiny custom signal object — either is fine; zero-dep is the only hard constraint. +- Test file placement follows the existing convention (API tests live under `apps/api/tests/`, never `src/`) — see `[[api-integration-test-db]]`. + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Phase scope & guarantees +- `.planning/ROADMAP.md` §"Phase 9: Faster Write-Back" — goal, the 5 Success Criteria, and the 3 owned pitfalls (no double-drain, create-before-delete under concurrent enqueues, hard constraints). + +### Existing implementation to extend (read before changing) +- `apps/api/src/broker/outboxWorker.ts` — `runOutboxDrain()` (the guarded drain), `isDraining` flag + its multi-process caveat comment, `startOutboxWorker()` (the 15s `setInterval`), edit-as-move ordering + durable CR-04 sibling-status gate, 412/hardfail/backoff/dead-letter handling. This is the path the signal must funnel into. +- `apps/api/src/routes/events.ts` — the three enqueue sites (create ~L300, edit/edit-as-move transaction ~L389–432, delete ~L511) where the post-commit signal must be published, and the sync-status endpoint the PWA polls. +- `apps/api/src/index.ts` §L136–141 — background-worker startup wiring (`startOutboxWorker` beside `startBrokerPoller`/`startReminderScheduler`); the trigger must be initialised consistently with this pattern, only under `isMainModule()`. + +No external specs/ADRs beyond ROADMAP.md — the durability decisions (D-04…D-12, CR-04/05, WR-01/04/06) are documented inline in `outboxWorker.ts` and `events.ts`. + + + + +## Existing Code Insights + +### Reusable Assets +- `runOutboxDrain()` + `isDraining` guard (`outboxWorker.ts`): the signal reuses this verbatim — no parallel drain path. The trailing re-drain (D-05) is a loop around it, not a second drainer. +- `startOutboxWorker()` `setInterval` pattern: the fallback stays; the new trigger sits alongside it and shares the same guarded entrypoint. +- Existing post-write `triggerTargetedResync` + `RESYNC_TIMEOUT_MS` flow already makes `status='done'` mean "local cache reflects the write" — unaffected by this phase; the signal only changes *when* the drain starts. + +### Established Patterns +- `setInterval`-only background workers (poller, reminderScheduler, outboxWorker) started exclusively under `isMainModule()` in `index.ts` — keeps timers out of the test process. The trigger follows the same gating. +- Per-uid exactly-once via the `pending AND next_attempt_at <= NOW()` selection + `isDraining`; the signal path must not introduce a second selection that could double-dispatch. + +### Integration Points +- Enqueue → signal: publish from `events.ts` after each `db.insert`/`db.transaction` commits (create, delete, and the move pair). +- Signal → drain: `outboxTrigger` listener sets `drainRequested` / invokes the guarded drain in `outboxWorker.ts`. +- Startup: initialise the trigger subscription in `index.ts` next to `startOutboxWorker()`. + + + + +## Specific Ideas + +The 1-2s target is "signal on enqueue, don't wait for the interval" — not a hard real-time budget. Actual latency is bounded by the single CalDAV round-trip to Fastmail, which is unchanged; this phase only removes the up-to-15s queueing delay before that round-trip begins. + + + + +## Deferred Ideas + +- **Multi-process / multi-replica outbox drain** — durable DB row-claim (`UPDATE calendar_outbox SET status='processing' WHERE id=? AND status='pending'`) instead of the in-memory `isDraining` guard. Documented in the `isDraining` declaration comment; out of scope while the deployment is single-process. Belongs to a future scaling phase, not v1.1. +- **Debounce/coalesce knob for burst write-back** — considered and rejected for now (D-06). Revisit only if Fastmail request-rate ever becomes a concern with more members (see `[[project-nmember-expansion]]`). + +None of the above are in scope for Phase 9. + + + +--- + +*Phase: 9-Faster-Write-Back* +*Context gathered: 2026-06-12* diff --git a/.planning/phases/09-faster-write-back/09-DISCUSSION-LOG.md b/.planning/phases/09-faster-write-back/09-DISCUSSION-LOG.md new file mode 100644 index 0000000..af65f65 --- /dev/null +++ b/.planning/phases/09-faster-write-back/09-DISCUSSION-LOG.md @@ -0,0 +1,59 @@ +# Phase 9: Faster Write-Back - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-06-12 +**Phase:** 9-Faster-Write-Back +**Areas discussed:** Burst handling, Verification, Fallback interval + +--- + +## Burst handling + +| Option | Description | Selected | +|--------|-------------|----------| +| Trailing re-drain (guarantee last edit) | If any signal arrives mid-drain, set `drainRequested` and re-run once the current drain finishes. Guarantees the LAST edit in a burst lands in ~1-2s. Slightly more CalDAV traffic under bursts. | ✓ | +| Short debounce/coalesce | Collect signals for a 200-500ms window and drain once. Gentler on Fastmail; last edit lands a fraction slower; adds a tunable knob. | | +| Fire-and-forget, rely on 15s for stragglers | Drop mid-drain signals; missed row waits for the next 15s tick. Simplest, but violates ~1-2s for the 2nd edit in a burst. | | + +**User's choice:** Trailing re-drain (guarantee last edit) +**Notes:** Two-user household — burst volume is tiny, so snappiness/correctness of the trailing edit outweighs Fastmail request-rate politeness. Debounce knob rejected as premature. + +--- + +## Verification + +| Option | Description | Selected | +|--------|-------------|----------| +| Automated integration test on the trigger | Vitest test asserting enqueue publishes the signal and `runOutboxDrain` fires promptly (mock CalDAV dispatch). Fast, CI-able, no live Fastmail. | ✓ | +| Both: integration test + operator stopwatch | Automated wiring test plus a one-time human checkpoint timing a real edit in the Fastmail native app. | | +| Operator stopwatch only | Human checkpoint timing real edit-to-Fastmail latency; nothing in CI guards regressions. | | + +**User's choice:** Automated integration test on the trigger +**Notes:** Dev bypass user (id 1) has no CalDAV credential/calendars, so live event-create 422s in the dev stack — an automated wiring test is the durable, CI-guarded evidence. Real wall-clock latency is bounded by the CalDAV round-trip and exercised in production use. + +--- + +## Fallback interval + +| Option | Description | Selected | +|--------|-------------|----------| +| Keep 15s | Leave the fallback at 15s exactly as today. Matches Success Criterion 5 wording; idle DB polling unchanged. | ✓ | +| Lengthen it (e.g. 30-60s) | Slow the fallback since the signal handles the hot path; fewer idle DB queries but slower recovery for missed rows; changes a number the criterion names. | | + +**User's choice:** Keep 15s +**Notes:** Idle DB polling cost is negligible for this single-process, two-user deployment. + +--- + +## Claude's Discretion + +- Exact shape of the `drainRequested`/re-drain loop (flag location, loop inside `runOutboxDrain` `finally` vs. scheduler wrapper) — left to planner/researcher, provided the no-double-drain guard and exactly-once guarantees hold. +- EventEmitter singleton vs. tiny custom signal object — either acceptable; zero-dependency is the only hard constraint. +- Test file placement per existing convention (`apps/api/tests/`, not `src/`). + +## Deferred Ideas + +- Multi-process / multi-replica outbox drain via durable DB row-claim — out of scope while single-process; future scaling phase. +- Debounce/coalesce knob for burst write-back — considered and rejected for now; revisit only if Fastmail request-rate becomes a concern with more members.