luckberg/familysync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: regenerate research SUMMARY for v1.2

Browse Source
This commit is contained in:
Lucas Berger
2026-06-19 14:01:13 -04:00
parent 4e0a5e38e8
commit 67430bb670
1 changed files with 79 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/research/SUMMARY.md
+78 -68
View File
@@ -1,105 +1,115 @@
# Project Research Summary
**Project:** FamilySync — v1.1 "Operability & Polish"
**Domain:** Self-hosted family calendar + lists PWA (operability/admin milestone on a shipped v1.0)
**Researched:** 2026-06-10
**Confidence:** HIGH (all findings grounded in direct v1.0 source inspection + v1.0 retrospective)
**Project:** FamilySync — v1.2 "Multi-Provider, Theming & Zero-Setup"
**Domain:** Self-hosted family calendar + lists PWA (opening beyond Fastmail to a second provider, on a shipped v1.1)
**Researched:** 2026-06-19
**Confidence:** HIGH for architecture/pitfalls (direct codebase inspection + shipped v1.0/v1.1 lessons); MEDIUM for Google Calendar API specifics (official docs via Context7); LOW for third-party UX patterns (cross-checked websearch)
## Executive Summary
v1.1 adds six operability/polish features to the proven v1.0 stack (Node 22 + Hono + Drizzle/MariaDB + tsdav/ical.js + web-push, React 19 + Vite + Schedule-X PWA, on Unraid/Docker behind Authelia OIDC + Pangolin/Newt, self-hosted Gitea with an Actions runner). The core stack is unchanged. Every feature reuses existing capabilities; **only one new dependency is warranted — `@playwright/test` (dev, `apps/pwa` scope)** for the authenticated mobile test harness. No new runtime packages: the setup wizard's validations are all covered by `zod` + `mysql2` + native `fetch` + `Buffer`/`web-push`.
v1.2 opens FamilySync beyond Fastmail. The keystone is a hand-rolled `CalendarProvider` TypeScript interface: the existing tsdav/ical.js CalDAV broker is refactored behind it **as a thin delegation wrapper with zero internal rewrite**, then Google Calendar (Google Calendar API v3 + OAuth2, not CalDAV) plugs in as a second implementation, and an in-memory mock provider plugs in as a third (dev-bypass + hermetic CI). On top of that seam: multiple reminders per event (1→N VALARMs / Google `overrides`), PWA dark mode (Light/Dark/System), and zero-manual-setup DB bootstrap (programmatic `migrate()` at boot). The milestone closes by applying CI-surfaced dependency updates.
The six features: (1) **per-event reminders** — VALARM authoring on the event form + a scheduler that honors each event's lead instead of the hardcoded 15-min; (2) **event-driven outbox drain** — cut perceived write-back latency from ~15s to ~1s; (3) **admin Settings** — role-gated UI to manage encrypted app passwords and designate the shared calendar; (4) **initial setup wizard** — first-run validated bootstrap of env/VAPID/DB/app-password; (5) **Gitea CI** — regression gate on PR + Docker image publish; (6) **mobile-emulated authed Playwright harness**.
The core stack is unchanged. **Two new runtime packages only** — `google-auth-library@10.7.0` and `@googleapis/calendar@15.0.0` (scoped, NOT the 50 MB `googleapis` monolith), both in `apps/api`. Dark mode, multiple reminders, migrate-on-boot, and dependency updates add **zero new dependencies** (Zustand `persist`, `ical.js` multi-VALARM, `drizzle-orm/mysql2/migrator`, and existing `pnpm` tooling already cover them).
Three preservation rules are non-negotiable and drive the design: VALARM authoring must **preserve native-client alarms on edit** (never rebuild-from-scratch and silently strip); the outbox durability guarantees (optimistic-202, create-before-delete ordering, drain concurrency guard, fresh-etag-before-PUT, per-uid exactly-once dedup) must be **unchanged** when the drain goes event-driven; and wizard-collected secrets (VAPID private key, `APP_PASSWORD_ENCRYPTION_KEY`) must **stay in env — never touch the DB or any response body**.
Three non-negotiable preservation rules drive the design: (1) the provider refactor must **not regress the live Fastmail path** — all v1.1 outbox/poller/scheduler integration tests must pass unchanged against the new interface; (2) the `reminder_lead_minutes`→`reminder_leads` JSON migration must **migrate existing data** before dropping the old column; (3) Google OAuth tokens are **per-member secrets**, AES-256-GCM encrypted, never in `app_config`, with the `member_credentials` unique constraint widened to `UNIQUE(user_id, provider_type)`.
## Key Findings
### Recommended Stack
No stack change. One new dev dependency; everything else reuses v1.0.
No stack change beyond two scoped Google packages (both `apps/api`):
**Core additions:**
- **`@playwright/test`** (dev, `apps/pwa`): the global `playwright-cli` binary is interactive tooling and exposes no `storageState`/`devices` presets — `@playwright/test` is required for CI spec files doing authenticated, device-emulated runs. The two coexist. Auth via the existing `DEV_AUTH_BYPASS` avoids mocking Authelia.
- **Gitea Actions workflows** (`.gitea/workflows/*.yml`, no npm packages): GitHub-Actions-compatible syntax but `runs-on: self-hosted`; job image `catthehacker/ubuntu:act-latest`; MariaDB service container `mariadb:11` with `healthcheck.sh --connect --innodb_initialized` (NOT `mysqladmin ping` — removed in MariaDB 11); Docker push via `docker/login-action@v3` + `docker/build-push-action@v5` needs a Gitea PAT with `write:package` scope (no built-in token has registry push rights).
- **Setup wizard validation — zero new deps:** env presence via `zod.safeParse`, DB via `mysql2` connect, VAPID via `Buffer.from(key,'base64url').length === 32`, OIDC via native `fetch('/.well-known/openid-configuration')`.
- **`google-auth-library@10.7.0`** — OAuth2 authorization-code flow + offline refresh-token management; ships its own types; `OAuth2Client` auto-refreshes expired access tokens via the `tokens` event.
- **`@googleapis/calendar@15.0.0`** — Google Calendar API v3 typed client (depends only on `googleapis-common`). **Reject the monolithic `googleapis`** (170+ clients, ~50 MB).
- **Provider abstraction** — hand-rolled TS interface in `apps/api/src/broker/`; no normalization library exists worth a dependency.
- **Dark mode** — pure CSS `[data-theme="dark"]` + Zustand `persist` (built into 5.0.14) + an inline `<script>` in `index.html`. No theming library.
- **Migrate-on-boot** — `drizzle-orm/mysql2/migrator` (already transitive); single connection, `multipleStatements: true`. **`drizzle-kit push` stays banned** on MariaDB 11.
- **Dependency updates** — existing `pnpm outdated`/`audit`; selective, pin-aware, one ecosystem at a time.
### Expected Features
**Must have (table stakes):**
**Table stakes:**
- Google events in the unified color-coded view (read sync), recurrence + all-day correct; per-calendar color extends the Phase 17 routing to Google calendar IDs.
- Cross-provider event create/edit/delete with a calendar picker that defaults to the member's most-used calendar and shows recognizable names (not provider IDs/emails).
- Self-service onboarding: Fastmail = app-password form with live CalDAV validation (reuse Phase 10 `CredentialSheet`); Google = single "Connect Google Calendar" OAuth button.
- One-tap reconnect on token expiry, surfaced as a **banner in the calendar view** (not buried in settings) — non-negotiable for the non-technical Apple member.
- Up to 5 reminders per event (add/remove rows, pre-populate all on edit), serialized to VALARMs / Google `overrides`, preserving other-client alarms.
- Light/Dark/System theme toggle in member Settings (per-device localStorage), no flash-of-wrong-theme, clean across every route incl. Schedule-X.
- Migrate-on-boot before `serve()`, idempotent, with a DB-readiness wait and fatal-on-failure.
- Mock provider seeded with recurring/all-day/timed/past/future + reminder events, wired to `DEV_AUTH_BYPASS` for hermetic CI.
- Per-event reminder selector with preset offsets (None / 5m / 10m / 15m / 30m / 1h / 2h / 1d / 2d); **"None" is the default** (no VALARM, no push). All-day events fire at 9 AM on the alert day (Apple convention). Existing VALARMs round-trip — never silently stripped.
- Admin Settings scoped to exactly two tasks: rotate/re-enter a member's app password (with inline CalDAV test) and toggle `calendars.is_shared`. Single `users.is_admin` boolean gate.
- Setup wizard: validated first-run steps (DB, app URL, OIDC, session secret auto-gen, encryption key auto-gen, VAPID auto-gen + structural check, admin account, Fastmail app password CalDAV PROPFIND test). Inline per-field validation; Next disabled until step passes.
- Faster write-back: target < 2s perceived; trigger an immediate drain on enqueue, keep the interval as fallback.
**Should have (competitive / differentiator):**
- Gitea CI PR gate + on-merge Docker publish.
- Mobile Playwright harness (`devices['iPhone 15']`, stored auth via `DEV_AUTH_BYPASS`).
- Multiple alarms per event (2× VALARM) — stretch, defer to v1.2.
**Anti-features (explicitly OUT — scope creep for a 2-member household):** notification-preferences UI, reminder snooze, wizard re-run, audit log, health dashboard, user management, provider abstraction (stays backlog 999.1), self-service member onboarding mixed into the wizard (stays backlog 999.5), real-device iOS CI.
**Anti-features (explicitly OUT):** two-way Fastmail↔Google mirroring, CalDAV-for-Google, service-account Google auth, tokens in cookies/localStorage, reminder templates/snooze, per-member server-synced theme, custom theme editor, app self-creating the database, Google per-event `colorId`, `googleapis` monolith.
### Architecture Approach
Findings grounded in the actual v1.0 codebase. Integration points (real paths):
Grounded in direct codebase inspection. Central seam: `broker/provider.ts` (`CalendarProvider` interface) + `broker/providerFactory.ts` (`createProvider(cred)` dispatching on `member_credentials.provider_type`: `caldav`/`google`/`mock`).
**Major components:**
- **CalDavProvider** wraps existing `client.ts`/`sync.ts`/`write.ts` **verbatim** — delegation, not refactor (Anti-Pattern 1).
- **poller.ts / outboxWorker.ts** become provider-agnostic: `createProvider(cred)` → `provider.listCalendars()`/`syncCalendar()`/`dispatchOutboxRow()`. The `isDraining` guard, `drainRequested` flag, optimistic-202, create-before-delete, and `uid:dtstartMs` dedup all stay in the worker (provider-agnostic) and must be unchanged.
- **GoogleCalendarProvider** — new; inline token refresh (check `expires_at` before each call, refresh within 5 min of expiry, retry-once on 401). Google OAuth callback mounted **pre-auth** (like Authelia `/callback`), at a distinct path with a distinct state cookie, redirect URI built from `EXTERNAL_BASE_URL` (never `req.host`).
- **Token storage** — research surfaces two options: a separate `provider_tokens` table (ARCHITECTURE.md) vs. extending `member_credentials` with `UNIQUE(user_id, provider_type)` (STACK/PITFALLS). Either way the unique constraint must widen to composite; `client_id`/`client_secret`/refresh tokens are env/encrypted, never in `app_config`. **Decide in the provider-abstraction phase.**
- **Multiple reminders** — `calendar_events.reminder_lead_minutes INT` → `reminder_leads JSON`, with a `JSON_ARRAY(...)` data-migration step before drop; `buildVeventString` loops N VALARMs; scheduler dedup key → `uid:dtstartMs:lead`; Google cap = 5 (reject 6+ before write).
- **Dark mode** — `themeStore.ts` (Zustand persist) + inline FOUC script in `index.html` + `[data-theme="dark"]` block mapping **all** `--sx-color-*` Schedule-X vars + dynamic `theme-color` meta for iOS.
- **Migrate-on-boot** — `runMigrationsIfNeeded()` in the `isMainModule()` guard, after the boot guards, before broker workers + `serve()`; `migration_lock` sentinel for concurrent-startup safety.
- **Mock provider** — `broker/mockProvider.ts`, in-memory, activated via `DEV_AUTH_BYPASS`/`provider_type='mock'`; resets on restart (intentionally stateless).
1. **DB migration** — `users.is_admin BOOLEAN DEFAULT 0`, `calendar_events.reminder_lead_minutes INT NULL`, an `app_config`/setup-state table. Foundation; blocks the role-gated and reminder work. (generate+migrate, never `push`.)
2. **Event-driven drain** — in-process `EventEmitter` (`lib/outboxTrigger.ts`, mirroring existing `listEmitter.ts`); signal after `db.insert(calendarOutbox)` in the three write handlers in `routes/events.ts`; subscribe in `startOutboxWorker()`. The existing `isDraining` guard already covers concurrent invocations. **Redis pub/sub is wrong here** — the drain is single-process by design.
3. **VALARM write path** — `buildVeventString` in `broker/vevent.ts` gains a `reminderMinutes?` param using `ICAL.Component('valarm')` + `ICAL.Duration.fromSeconds` (same ical.js surface as RRULE). `eventFieldsSchema` (routes/events.ts) and `outboxPayloadSchema` (outboxWorker.ts) must change in sync (flagged by the IN-03 comment).
4. **Variable-lead scheduler** — `reminder_lead_minutes` populated by `sync.ts` parsing the VALARM TRIGGER; scheduler query uses `DATE_SUB(dtstart_utc, INTERVAL reminder_lead_minutes MINUTE)` over a ±1-min window; dedup key becomes compound `uid:dtstartMs`. Drop the `isShared`-only restriction for reminder pushes (a user who set an alarm wants it regardless of calendar).
5. **Admin role + setup wizard** — `routes/admin.ts` with `requireAdmin` middleware reusing `broker/crypto.ts`; setup wizard and admin Settings are two frontend consumers of the same `/api/admin/*` + `/api/setup/*` routes (do not duplicate). `GET /api/setup/status` mounts **before** the OIDC guard (like `/health`). VAPID/AES keys stay in env — wizard generates + displays for the operator to copy.
### Recommended Phase Structure (dependency-ordered, continues from v1.1 Phase 20 → starts at Phase 21)
### Recommended Phase Structure (dependency-ordered)
ARCHITECTURE.md's authoritative ordering:
Starting at **Phase 7** (continues v1.0 numbering). Critical path with two fully independent parallel tracks:
1. **Phase 21 — Zero-Setup DB Bootstrap.** `runMigrationsIfNeeded()` + `migration_lock` sentinel. No deps; unlocks every later phase to assume auto-migrate. Risk: low.
2. **Phase 22 — Provider Abstraction (CalDAV only, Fastmail unchanged).** Interface + factory + `CalDavProvider` wrapper; poller/outbox use `createProvider`. Dep: 21. **Risk: HIGH** — must not regress Fastmail; golden-path integration tests pass unchanged. Includes the `UNIQUE(user_id, provider_type)` migration.
3. **Phase 23 — Multiple Reminders.** `reminder_leads` JSON + data-migration; multi-VALARM; scheduler multi-lead + dedup-key change; form add/remove. Dep: 22. Risk: MEDIUM (breaking migration).
4. **Phase 24 — Mock Provider (dev/CI).** Seeded in-memory provider; factory `mock` case; Playwright exercises calendar CRUD. Dep: 22. Risk: low.
5. **Phase 25 — Google Calendar Provider.** Token storage + `googleProvider.ts` + OAuth authorize/callback routes + pre-auth mount. Deps: 22, 23 (reminder serialization), 21 (tokens table). Risk: MEDIUM (OAuth edge cases, recurrence/sync model).
6. **Phase 26 — Self-Service Onboarding UI.** `/api/me/providers` GET/DELETE, `ProviderConnectSheet`, reconnect banner. Dep: 25. Risk: low.
7. **Phase 27 — PWA Dark Mode.** FOUC script, dark tokens, `themeStore`, `ThemeToggle`. No deps — **parallelizable with 22–26**. Risk: low.
8. **Phase 28 (or folded) — Dependency Updates (DEP-01).** Last; per-ecosystem batches with CI green after each. Risk: low but cascade-prone (ESLint 9.x pin, Schedule-X hold, Drizzle coupling).
1. **DB foundation** (migration: is_admin, reminder_lead_minutes, setup-state) — blocks the role/reminder work.
2. **Faster write-back** (event-driven drain) — small, low-risk, immediate benefit; independent after migration.
3. **Admin role + routes** — precondition for both admin Settings and setup wizard.
4. **Setup wizard** (backend + UI) — depends on the admin/role + config plumbing.
5. **Admin Settings UI** — depends on admin role; shares routes with the wizard.
6. **Per-event reminders** (VALARM authoring + variable-lead scheduler) — independent track, largest scope; keep authoring + preserve-on-edit + scheduler in one phase.
7. **Gitea CI** — fully independent; start with a runner-probe step.
8. **Mobile Playwright harness** — fully independent.
Tracks 7 and 8 have no code dependencies and can run parallel to anything.
`THEME-01` (27) and `SETUP-05` (21) are independent of the multi-provider chain. The roadmapper should confirm whether dark mode and dependency updates are standalone phases or folded.
### Critical Pitfalls (phase-mapped)
1. **VALARM round-trip strips native alarms** (reminders phase) — update path must extract + preserve existing VALARM from `rawVevent`, not rebuild from scratch.
2. **TRIGGER serialized as VALUE=TEXT** (reminders phase) — use `ICAL.Duration.fromSeconds(-n*60)`, verify ICS has no `VALUE=TEXT`.
3. **All-day reminder semantics undefined** (reminders phase) — disable selector when `allDay` in UI; guard API.
4. **Per-event lead breaks uid-only dedup** (reminders phase) — rescheduled events go invisible; fix with `uid:dtstartMs` compound key.
5. **Event-driven drain double-execution** (write-back phase) — use a `drainRequested` flag checked by the interval callback, not direct concurrent `runOutboxDrain()` calls; preserves create-before-delete + etag handling.
6. **Setup endpoint reachable post-setup** (wizard phase) — guard checked on every invocation (member_credentials AND VAPID env present → 423), not just at startup; never log/echo the app password; secrets never persisted to DB.
7. **Gitea ghost failures** (CI phase) — MariaDB readiness race (healthy ≠ accepting connections) and runner env assumptions (Node 22 + pnpm not guaranteed) — start with a probe-only workflow.
1. **Provider refactor regresses the live Fastmail path** (P22) — wrong `provider_type` branch silently stops all write-back; keep `caldav` the default, change zero existing tests.
2. **Google refresh tokens expire after 7 days in "Testing" publishing status** (P25) — use the non-restricted `calendar.readonly` + `calendar.events` scopes + Production status to avoid Google verification; add both accounts as test users; store grant timestamp.
3. **Google recurrence ≠ iCalendar RRULE** (P25) — fetch with `singleEvents: false`, fetch modified/`cancelled` instances separately, map cancellations to EXDATE, reconstruct a VCALENDAR string for the existing `expand.ts`.
4. **Google `syncToken` 410 Gone halts sync + full re-sync overwrites un-drained local writes** (P25) — catch 410 → drop token → enqueue full re-sync; drain outbox (or skip pending UIDs) before upsert.
5. **Google all-day/timezone differs** (P25) — normalize `start.dateTime` to UTC `...Z` (never a TZID without VTIMEZONE); map `reminders.useDefault:true` → null ("no FamilySync reminder").
6. **Multiple VALARM dedup + Google 5-cap** (P23/P25) — dedup key `uid:dtstartMs:lead`; preserve-vs-replace driven by a `remindersChanged` flag; reject 6+ reminders before a Google write.
7. **Drizzle auto-migrate on MariaDB 11 — three failure modes** (P21) — journal-hash mismatch, false destructive diff (only if `push` is misused — use the `migrator` API), concurrent-startup race (the `migration_lock` sentinel); log each applied migration.
8. **Dark mode FOWT + Schedule-X coverage + SW cache** (P27) — inline pre-paint script; override **all** `--sx-color-*`; keep theme CSS in hashed build output.
9. **OAuth through Pangolin — redirect_uri_mismatch + state-cookie collision** (P25/26) — distinct callback path outside the Authelia middleware scope, distinct `google.oauth.state` cookie, redirect URI from `EXTERNAL_BASE_URL`.
10. **Token storage — encryption-key reuse + `UNIQUE(user_id)` blocks multi-provider** (P22/25) — widen to `UNIQUE(user_id, provider_type)`; persist only the encrypted refresh token, cache access tokens in memory; minimum scopes.
11. **Dependency-update breaking cascade** (P28) — ESLint held at 9.x (eslint-plugin-react), Schedule-X minor-only until dark mode locked, Drizzle orm+kit bumped together; never bulk `pnpm up`/`audit --fix`.
## Research Flags (deeper investigation during planning)
- **Reminders phase:** confirm Fastmail CalDAV accepts the chosen TRIGGER value type; exact `rawVevent` round-trip on the update path.
- **Wizard phase:** OIDC discovery failure handling (retry/timeout/fallback).
- **CI phase:** probe the actual Unraid Gitea runner (Docker socket mount, Node/pnpm versions) before designing pipelines.
- **Mobile harness:** fixed user (DEV_AUTH_BYPASS user 1) vs parameterized; prod service worker must be neutralized in the base context.
- **Provider phase:** token-storage shape (`provider_tokens` table vs. extended `member_credentials`) — pick one in P22; verify the `migrator` API behavior against the installed `drizzle-orm` version before shipping auto-migrate.
- **Google phase:** confirm the exact scope tier needed for write-back without Google verification; end-to-end OAuth test over the Pangolin hostname (not localhost); iOS-Safari standalone callback is load-bearing (human gate, like Phase 3).
- **Reminders phase:** confirm the `remindersChanged`/preserve-on-edit contract end-to-end; index the reminder column for the scheduler scan.
- **Dark mode:** enumerate every `--sx-color-*` from `@schedule-x/theme-default`; verify it's in the Workbox precache (hashed).
**Well-documented — skip research:** event-driven drain (EventEmitter proven in `listEmitter.ts`), admin/wizard routes (standard Hono + zod + React forms), Playwright device/storageState APIs.
## Open Questions to Resolve in Requirements / Roadmap
## Open Questions to Resolve in Requirements
1. Admin tab visibility — both members are operators; should the non-technical member see it?
2. Drain transport — confirm single-container deployment (direct in-process signal) vs any multi-replica need (would require Redis).
3. Playwright auth — DEV_AUTH_BYPASS-only vs programmatic OIDC; fixed vs parameterized user.
1. Are **dark mode (27)** and **dependency updates (28)** standalone phases or folded into adjacent work? (Both are independent of the provider chain.)
2. Token storage: separate `provider_tokens` table vs. extended `member_credentials` — resolve in P22.
3. Google all-day "9 AM local" reminder semantics can't be replicated (Google fires midnight-minus-lead) — accept as a documented per-provider difference.
## Confidence
| Domain | Confidence | Notes |
| ---------------- | ---------- | ---------------------------------------------------------------------------------------- |
| Stack | HIGH | v1.0 proven; one dev dep; Gitea syntax compatible (MariaDB 11 healthcheck caveat noted) |
| Features | HIGH | All have prior art; scoped to a tiny household |
| Architecture | HIGH | Grounded in real v1.0 source; component boundaries + build order sound |
| Pitfalls | HIGH | 15 pitfalls from codebase review + RFC 5545 + v1.0 retrospective, each mapped to a phase |
| Unraid CI runner | MEDIUM | Runner Docker-socket/Node/pnpm state unknown until probed |
| ------ | ---------- | ----- |
| Stack | HIGH | Two scoped Google packages verified 2026-06-19; everything else reuses shipped deps |
| Features | MEDIUM | Google API via official docs; UX patterns cross-checked but LOW-confidence sources |
| Architecture | HIGH | Grounded in direct v1.1 source; component boundaries + build order sound |
| Pitfalls | HIGH | From codebase review + Google API behavior + v1.0/v1.1 migration/CI lessons, each phase-mapped |
| Google OAuth through Pangolin | MEDIUM | Redirect-URI + state-cookie mechanics sound; needs live end-to-end validation |
## Sources
See `STACK.md`, `FEATURES.md`, `ARCHITECTURE.md`, and `PITFALLS.md` in this directory for full citations (Google Calendar API docs, google-auth-library, Drizzle migrator, Schedule-X theming, pnpm audit/outdated, family-calendar UX surveys, and direct codebase inspection of `apps/api/src/broker/`).
---
_Synthesized for: FamilySync — v1.2 Multi-Provider, Theming & Zero-Setup_
_Researched: 2026-06-19_