familysync/.planning/milestones/v1.1-phases/09-faster-write-back/09-CONTEXT.md

# Phase 9: Faster Write-Back - Context

**Gathered:** 2026-06-12
**Status:** Ready for planning

<domain>
## Phase Boundary

Make a created/edited/deleted event reach Fastmail in ~1-2s instead of waiting up to ~15s for the next interval tick, by signalling the outbox drain on enqueue (event-driven) — with every existing outbox durability guarantee intact.

**In scope:** the in-process signal mechanism, its wiring into the existing drain path, and a trailing re-drain guarantee under bursts.

**Out of scope (locked by roadmap — do NOT reopen):**
- Redis for the drain (Redis stays only for list SSE). Single-process, single Node module instance by design.
- node-cron (silently skipped ticks in the long-lived process — `setInterval` only). See `[[node-cron-skips-in-long-running-process]]`.
- Any change to the route handlers' optimistic-202 / no-inline-CalDAV contract.
- Any change to outbox durability semantics (fresh-etag-before-PUT, 412 conflict flow, per-uid exactly-once, edit-as-move create-before-delete, dead-letter/backoff).
- Multi-process / multi-replica row-claim (the `isDraining` declaration comment documents the future `UPDATE ... status='processing'` path; not this phase).

</domain>

<decisions>
## Implementation Decisions

### Signal mechanism (locked from ROADMAP.md — restated so the planner doesn't re-derive)
- **D-01:** New artifact is a single zero-dependency in-process `EventEmitter` at `apps/api/src/lib/outboxTrigger.ts`. No new npm dependency.
- **D-02:** The signal funnels through the **existing** `isDraining`-guarded `runOutboxDrain()` path via a `drainRequested` flag. The trigger must NEVER call `runOutboxDrain()` directly in a way that bypasses the `isDraining` guard or escapes the error-caught wrapper (Pitfall 5 — no double-drain).
- **D-03:** The signal is published **after** the enqueue transaction commits — never between the two inserts of an edit-as-move (Pitfall 6). For a move, publish once after both CREATE and DELETE rows are committed, so create-before-delete ordering is never raced by the signal.
- **D-04:** Route handlers still return optimistic 202 immediately and make no inline CalDAV call; the signal is fire-and-forget.

### Burst behavior — trailing re-drain (the main decision this discussion locked)
- **D-05:** **Guarantee the last edit in a burst.** When a signal arrives while a drain is already in flight, set `drainRequested` and re-run the drain exactly once after the current one finishes (drain-again-if-requested loop), so the LAST edit in a rapid burst still lands in ~1-2s rather than waiting for the next 15s tick.
- **D-06:** No debounce/coalesce window and no rejection of mid-drain signals. Rationale: two-user household — burst volume is tiny; correctness/snappiness of the trailing edit outweighs Fastmail request-rate politeness. (A debounce knob was considered and rejected as premature.)
- **D-07:** Implementation must preserve exactly-once per uid when the signal and the 15s fallback overlap — the trailing re-drain reuses the same `pending AND next_attempt_at <= NOW()` selection and `isDraining` guard, so no duplicate CalDAV PUT for the same row (Success Criterion 4).

### Fallback interval
- **D-08:** **Keep the 15s `setInterval` fallback exactly as-is.** It still runs for startup catch-up and transient-error recovery (Success Criterion 5). Do not lengthen it — idle DB polling cost is negligible for this deployment and the criterion names 15s explicitly.

### Verification
- **D-09:** Prove Success Criterion 1 with an **automated integration test on the trigger wiring** (Vitest): assert that enqueuing a row publishes the signal and that `runOutboxDrain` is invoked promptly (stub/mock the CalDAV dispatch so no live Fastmail is needed). Also cover the trailing-re-drain case (signal during in-flight drain → exactly one follow-up drain) and the no-double-PUT overlap case.
- **D-10:** No operator-stopwatch human checkpoint is required for this phase. Rationale: the dev bypass user (id 1) has no CalDAV credential/calendars, so live event-create 422s in the dev stack — see `[[dev-data-user1-no-calendars]]`; an automated wiring test is the durable, CI-guarded evidence. (Real wall-clock-to-Fastmail latency is bounded by the CalDAV round-trip regardless and is implicitly exercised in production use.)

### Claude's Discretion
- Exact shape of the `drainRequested`/re-drain loop (where the flag lives, whether the loop sits inside `runOutboxDrain`'s `finally` or in the scheduler wrapper) — planner/researcher decides, provided D-02/D-05/D-07 hold.
- Whether the EventEmitter is a module singleton vs. a tiny custom signal object — either is fine; zero-dep is the only hard constraint.
- Test file placement follows the existing convention (API tests live under `apps/api/tests/`, never `src/`) — see `[[api-integration-test-db]]`.

</decisions>

<canonical_refs>
## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Phase scope & guarantees
- `.planning/ROADMAP.md` §"Phase 9: Faster Write-Back" — goal, the 5 Success Criteria, and the 3 owned pitfalls (no double-drain, create-before-delete under concurrent enqueues, hard constraints).

### Existing implementation to extend (read before changing)
- `apps/api/src/broker/outboxWorker.ts` — `runOutboxDrain()` (the guarded drain), `isDraining` flag + its multi-process caveat comment, `startOutboxWorker()` (the 15s `setInterval`), edit-as-move ordering + durable CR-04 sibling-status gate, 412/hardfail/backoff/dead-letter handling. This is the path the signal must funnel into.
- `apps/api/src/routes/events.ts` — the three enqueue sites (create ~L300, edit/edit-as-move transaction ~L389–432, delete ~L511) where the post-commit signal must be published, and the sync-status endpoint the PWA polls.
- `apps/api/src/index.ts` §L136–141 — background-worker startup wiring (`startOutboxWorker` beside `startBrokerPoller`/`startReminderScheduler`); the trigger must be initialised consistently with this pattern, only under `isMainModule()`.

No external specs/ADRs beyond ROADMAP.md — the durability decisions (D-04…D-12, CR-04/05, WR-01/04/06) are documented inline in `outboxWorker.ts` and `events.ts`.

</canonical_refs>

<code_context>
## Existing Code Insights

### Reusable Assets
- `runOutboxDrain()` + `isDraining` guard (`outboxWorker.ts`): the signal reuses this verbatim — no parallel drain path. The trailing re-drain (D-05) is a loop around it, not a second drainer.
- `startOutboxWorker()` `setInterval` pattern: the fallback stays; the new trigger sits alongside it and shares the same guarded entrypoint.
- Existing post-write `triggerTargetedResync` + `RESYNC_TIMEOUT_MS` flow already makes `status='done'` mean "local cache reflects the write" — unaffected by this phase; the signal only changes *when* the drain starts.

### Established Patterns
- `setInterval`-only background workers (poller, reminderScheduler, outboxWorker) started exclusively under `isMainModule()` in `index.ts` — keeps timers out of the test process. The trigger follows the same gating.
- Per-uid exactly-once via the `pending AND next_attempt_at <= NOW()` selection + `isDraining`; the signal path must not introduce a second selection that could double-dispatch.

### Integration Points
- Enqueue → signal: publish from `events.ts` after each `db.insert`/`db.transaction` commits (create, delete, and the move pair).
- Signal → drain: `outboxTrigger` listener sets `drainRequested` / invokes the guarded drain in `outboxWorker.ts`.
- Startup: initialise the trigger subscription in `index.ts` next to `startOutboxWorker()`.

</code_context>

<specifics>
## Specific Ideas

The 1-2s target is "signal on enqueue, don't wait for the interval" — not a hard real-time budget. Actual latency is bounded by the single CalDAV round-trip to Fastmail, which is unchanged; this phase only removes the up-to-15s queueing delay before that round-trip begins.

</specifics>

<deferred>
## Deferred Ideas

- **Multi-process / multi-replica outbox drain** — durable DB row-claim (`UPDATE calendar_outbox SET status='processing' WHERE id=? AND status='pending'`) instead of the in-memory `isDraining` guard. Documented in the `isDraining` declaration comment; out of scope while the deployment is single-process. Belongs to a future scaling phase, not v1.1.
- **Debounce/coalesce knob for burst write-back** — considered and rejected for now (D-06). Revisit only if Fastmail request-rate ever becomes a concern with more members (see `[[project-nmember-expansion]]`).

None of the above are in scope for Phase 9.

</deferred>

---

*Phase: 9-Faster-Write-Back*
*Context gathered: 2026-06-12*