luckberg/familysync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: complete project research (stack, features, architecture, pitfalls, summary)

Browse Source
This commit is contained in:
Lucas Berger
2026-06-03 15:01:33 -04:00
parent 1ce0348914
commit 0f79277942
5 changed files with 1740 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/research/SUMMARY.md
+278
View File
@@ -0,0 +1,278 @@
# Project Research Summary
**Project:** FamilySync
**Domain:** Self-hosted family calendar + shared-lists PWA (Fastmail-brokered, Authelia-authed, Unraid-hosted)
**Researched:** 2026-06-03
**Confidence:** MEDIUM-HIGH overall; LOW on personal-calendar CalDAV ACL behavior (the critical unresolved risk)
## Executive Summary
FamilySync is a purpose-built two-person household coordination hub: a React PWA that aggregates Fastmail-hosted calendars (shared family + each member's personal) into a unified color-coded view, adds shared collaborative lists with live co-edit sync, and delivers Web Push alerts — all behind Authelia OIDC with zero app-store friction. The project has resolved its JMAP-vs-CalDAV open question: Fastmail does not expose calendars over JMAP. CalDAV via `tsdav` is the only available protocol and the one to build against exclusively. The broker-cache architecture (background ctag polling + MariaDB cache) is mandatory because Fastmail provides no push webhook; all calendar reads hit the local cache, not Fastmail on-demand.
The single biggest v1 risk is whether the broker's single app password can discover and read the wife's personal Fastmail calendar after it has been shared via Fastmail's in-app CalDAV ACL flow. Fastmail's documentation confirms the sharing feature exists and follows CalDAV draft ACL standards, but does not explicitly state whether shared calendars appear in the sharing account's principal discovery. This must be spiked and confirmed in Phase 1, before any work is committed to the personal-calendar overlay feature. The fallback is per-member app passwords (at most two credentials for this household). A deeper risk: if the wife's personal calendar lives on iCloud rather than Fastmail, the broker cannot read it at all — this must be confirmed as part of the same spike. Canada is unaffected by the EU DMA PWA restriction; iOS push applies to both household members without the DMA caveat.
iOS Web Push has multiple silent failure modes that are not obvious until production: the PWA must be installed to Home Screen (no browser-tab push on iOS), `showNotification()` must be wrapped in `event.waitUntil()` or iOS permanently revokes the subscription after three silent pushes, and Apple's ITP can silently clear service worker registrations. Recurring events are the highest-complexity table-stakes feature — use CalDAV server-side `CALDAV:expand` in REPORT requests to get pre-expanded instances rather than relying solely on client-side RRULE expansion, and defer single-instance RECURRENCE-ID edits to v1.x. The recommended stack has strong convergence across all four research threads: Hono + Drizzle (mysql2/MariaDB) + tsdav + ical.js + rrule + vite-plugin-pwa + web-push + @hono/oidc-auth, with SSE (preferred over WebSockets for proxy resilience through Pangolin) + optional Redis for list sync.
## Key Findings
### Recommended Stack
The stack is tightly constrained by the existing infra (MariaDB, no PostgreSQL; Unraid Docker Compose; Authelia OIDC; Pangolin tunnel) and the Fastmail CalDAV protocol decision. All four research files converged on the same library choices with no meaningful disagreement.
**Core technologies:**
- **Hono 4.x + @hono/node-server** — HTTP framework; Web Standards-native, first-class TypeScript, built-in `streamSSE` helper for SSE, WebSocket via node adapter. Lighter than Express; better TypeScript than Fastify for this size.
- **Drizzle ORM 0.45.x + mysql2 3.x** — Type-safe MariaDB access; no binary engine (unlike Prisma); wire-compatible with MariaDB via `mysql` dialect; `drizzle-kit` for migrations.
- **tsdav 2.2.2** — The only maintained TypeScript CalDAV client; handles PROPFIND, REPORT, PUT, DELETE against Fastmail's `caldav.fastmail.com` endpoint.
- **ical.js 2.2.1** — Mozilla-maintained iCalendar parser; parse raw VCALENDAR/VEVENT payloads returned by tsdav; handles VTIMEZONE, RDATE, EXDATE.
- **rrule 2.8.1** — RRULE string expansion for UI display layer (calendar window ±1 month); use only after confirming against server-side CALDAV:expand for authoritative instances.
- **vite-plugin-pwa 1.3.0 (injectManifest strategy)** — Workbox integration for service worker; use `injectManifest` not `generateSW` for explicit SW control required by Web Push subscription management.
- **web-push 3.6.7** — VAPID push sender; handles APNs Web Push (iOS) and FCM (Android); design payloads as self-contained Declarative Web Push JSON from day one (ITP-resilient on iOS 18.4+).
- **@hono/oidc-auth 1.8.3** — Storage-less JWT session cookies; authorization-code + PKCE against Authelia's OIDC endpoint. Do NOT use `oidc-client-ts` (browser-side SPA library; wrong layer).
- **React 19 + TanStack Query 5 + Zustand 5** — TanStack Query owns all server state; Zustand owns pure UI state (selected date, color assignments, drawer state).
- **SSE via Hono `streamSSE`** — Preferred over WebSockets for list sync; unidirectional server-to-client, more proxy-resilient through Pangolin. Verify Pangolin WS pass-through in an infra spike if WS is chosen instead.
- **ioredis 5.x** — Redis pub/sub for broadcasting list-change events; optional for single-container deployment (in-process EventEmitter works for one Node process).
**Critical decisions locked:**
- CalDAV only, not JMAP — no reconsideration
- Principal URL: `https://caldav.fastmail.com/dav/principals/user/{email}/` — bare root is insufficient
- IANA timezone IDs only (e.g., `America/Toronto`) — never Windows-style IDs in emitted VTIMEZONE
- Serve `sw.js` with `Cache-Control: no-cache` — Safari respects HTTP cache for SW scripts
### Expected Features
**Must have (table stakes — v1):**
- Unified multi-calendar view with per-member color coding (shared family + personal calendars)
- Day / week / month / agenda views
- Create / edit / delete events with CalDAV write-back (all-day and timed)
- Recurring events: display and create (via CALDAV:expand); single-instance edit deferred to v1.x
- Event push notifications (reminders) and list-change notifications
- Shared lists: create named lists, add/check/reorder/delete items
- Live list co-edit sync (SSE or WebSocket)
- OIDC login via Authelia — must feel seamless for non-technical user
- PWA installability (Add to Home Screen) with guided iOS install UX — load-bearing, not optional
**Should have (differentiators — v1):**
- Personal calendar overlay (shared + each member's personal in one view) — THE differentiator; blocked on CalDAV ACL spike
- Event change push notifications ("partner changed an event") — documented Google Family Calendar gap
- Optimistic list check-off with instant feedback
**Defer (v2+):**
- Wall-display / kiosk dashboard — explicitly out of scope per PROJECT.md
- Single-instance recurring event edit (RECURRENCE-ID) — high complexity, low v1 priority; ship "edit all instances" only with clear label
- "This and following" recurring edit
- Timezone display toggle
- Meal planning, chore system, AI import, RSVP, event comments — confirmed anti-features for this household
### Architecture Approach
The architecture is a single Docker Compose stack on Unraid: one Node/Hono API container (broker + list + push + scheduler in one process), MariaDB, and optional Redis. The broker pattern is mandatory: all calendar reads hit the MariaDB cache; a background ctag poller (5-min interval) issues lightweight PROPFIND to detect changes and REPORT to sync. No Fastmail push webhook exists. The PWA serves as the sole frontend; Authelia handles all authentication upstream via Pangolin tunnel. Pangolin/Newt WebSocket pass-through must be verified in an infrastructure spike before real-time sync is built.
**Major components:**
1. **CalDAV Broker** (`broker/`) — tsdav wrapper, ctag poller, REPORT sync, RRULE expand, MariaDB cache write. All Fastmail I/O isolated here; nothing else imports from this module.
2. **App API** (Hono) — OIDC session middleware, REST routes for calendars/events/lists, SSE hub, push subscription routes.
3. **Reminder Scheduler** — In-process node-cron; queries events within 15-min window, fires VAPID push.
4. **Lists Domain** — REST CRUD + Redis pub/sub emit on every write; SSE hub delivers to connected clients.
5. **React PWA** — Calendar views (rrule.js for display expansion), list co-edit UI, push subscription registration, guided iOS install flow.
6. **MariaDB** — Persistent store: users, calendar event cache (raw VEVENT blob + dtstart_utc + ctag), lists/items, push_subscriptions.
**Key patterns:**
- Broker cache + ctag polling — never proxy calendar reads to Fastmail on-demand
- Store raw VEVENT blob + `dtstart_utc` column (DATE type for all-day) — never pre-expand RRULE into rows
- Write-through cache invalidation — after CalDAV PUT, re-fetch server's version; never cache the client's version
- Use `oidc_iss + oidc_sub` as stable identity composite key — never `email`
- All-day events: `{ date: "YYYY-MM-DD", allDay: true }` throughout the stack — never coerce to JS Date or SQL DATETIME
### Critical Pitfalls
1. **Personal calendar CalDAV ACL (LOW confidence — must spike in Phase 1)** — Broker token may not discover the wife's personal calendar after sharing unless Fastmail's share+accept flow causes it to appear in principal discovery. Additionally, the wife's personal calendar may be on iCloud, making broker access impossible. Spike before committing to the overlay feature. Fallback: per-member app passwords (two credentials max).
2. **All-day events stored as DATETIME** — `DATE` (not `DATETIME`) in iCalendar; storing as UTC shifts events to the previous day in negative-offset timezones. Store as `DATE` column in MariaDB; emit ISO date string without time component; detect `allDay: true` in the frontend. This bug has hit Home Assistant's CalDAV integration in production.
3. **iOS push subscription silent revocation** — iOS permanently revokes push subscriptions after 3 silent pushes (push event received but no visible notification). Always wrap `showNotification()` in `event.waitUntil()`. Payloads must be self-contained (no fetch-before-display). Implement subscription health-check on every app open. Handle 410 Gone immediately.
4. **Recurring event RECURRENCE-ID write corruption** — Editing a single occurrence requires injecting a new VEVENT block with RECURRENCE-ID into the same VCALENDAR resource. For v1, restrict to "edit all instances" only with a clear label; defer single-instance override to v1.x.
5. **Pangolin WebSocket pass-through** — Known issue (#1034) where WebSocket connections fail through Pangolin even when HTTP works. Verify SSE or WS pass-through in an infra spike before building real-time list sync. SSE is more proxy-resilient and is the preferred transport.
6. **ETag / sync-token incremental sync** — Use WebDAV-Sync (sync-token) for delta fetches from day one; fall back to ctag polling if sync-token returns 403. Include `If-Match: <etag>` on CalDAV PUT; handle 412 (concurrent modification).
7. **iOS install guide is load-bearing** — iOS has no browser install prompt. The in-app guided flow (annotated screenshots: share icon → Add to Home Screen) is the only path to PWA installation and therefore push notifications for the wife.
## Implications for Roadmap
Suggested build order based on the dependency graph across all four research files:
### Phase 1: Foundation + CalDAV Broker Spike
**Rationale:** Nothing works without auth. The CalDAV broker is the riskiest unknown (personal calendar ACL) and must be validated before any calendar UI is built. Pangolin SSE/WS pass-through also must be confirmed before real-time sync is designed. Both are binary go/no-go decisions.
**Delivers:** Working Docker Compose stack; Authelia OIDC login; CalDAV broker read path confirmed against real Fastmail account; personal-calendar ACL spike result (go/no-go on the overlay feature); wife's calendar location confirmed (Fastmail vs iCloud); Pangolin SSE pass-through verified.
**Addresses:**
- OIDC login — required by all other features
- CalDAV protocol confirmed (JMAP ruled out)
- Personal calendar sharing ACL confirmed or fallback decided
- Infrastructure scaffold (Docker Compose, MariaDB schema, drizzle-kit migrations, .env)
**Pitfalls to avoid:**
- JMAP assumption — confirm CalDAV endpoint returns events before proceeding
- Email as identity key — use `oidc_iss + oidc_sub` from schema day one
- Authelia groups claim — skip groups; all authenticated users are equal in this household
- Pangolin WebSocket — smoke test SSE over public URL in this phase
**Research flag:** NEEDS research-phase — CalDAV ACL sharing mechanics and Pangolin WebSocket behavior are both implementation-dependent; cannot be resolved from documentation alone.
---
### Phase 2: Calendar Display
**Rationale:** Build read-only calendar UI on top of the confirmed broker. Validates broker correctness before adding write complexity. Color coding, multi-view layout, and RRULE display are substantial enough to be their own phase.
**Delivers:** Unified calendar view (month/week/day/agenda); per-member color coding; all-day event display; recurring event display via CALDAV:expand; no write-back yet.
**Addresses:**
- Unified multi-calendar view
- Per-member color coding
- Day / week / month / agenda views
- Recurring event display (RRULE + EXDATE)
**Pitfalls to avoid:**
- All-day events as DATETIME — `allDay: true` flag + `DATE` column, no timezone coercion
- RRULE client-side expansion only — use CALDAV:expand in REPORT; rrule.js for display window only
- DST shift on recurring events — test with events crossing spring/fall DST boundary before milestone sign-off
- Service worker cache-first for API routes — `Network-First` for `/api/*` routes
**Research flag:** Standard patterns; no research phase needed.
---
### Phase 3: Event Write-Back + PWA Install
**Rationale:** Adds create/edit/delete on top of confirmed read path. PWA installability is paired here because the service worker is required for both offline shell caching and push registration in Phase 5.
**Delivers:** Full CRUD events written back to Fastmail via CalDAV PUT; all-day and timed events; guided iOS Add to Home Screen onboarding flow; PWA manifest + service worker (injectManifest strategy).
**Addresses:**
- Create / edit / delete events
- PWA installability
- iOS install guide (load-bearing for wife's push notifications)
**Pitfalls to avoid:**
- Write-back without `If-Match` ETag — include on all PUT/DELETE; handle 412
- Recurring RECURRENCE-ID corruption — v1 ships "edit all instances" only with clear label
- Write-through cache — after PUT, re-fetch server version before caching
- SW served with aggressive cache headers — `Cache-Control: no-cache` on `sw.js` and manifest
- SW update staleness — in-app "new version available" prompt on SW `waiting` state
**Research flag:** Standard patterns; no research phase needed.
---
### Phase 4: Shared Lists + Live Sync
**Rationale:** Lists domain is architecturally independent of the calendar broker (shares only auth and MariaDB). Serialized here to reduce WIP; could parallel-track with Phase 3 after Phase 1 is complete.
**Delivers:** Named shared lists; item CRUD (add/check/reorder/delete); live co-edit sync via SSE + optional Redis pub/sub; optimistic check-off; list sequence number for reconnect replay.
**Addresses:**
- Shared lists CRUD
- Live list co-edit sync
- Reconnect gap handling
**Pitfalls to avoid:**
- Missed updates during reconnect — monotonic sequence number per list in initial schema; server replays on reconnect
- Pangolin SSE timeout — verify long-lived connections survive; configure proxy timeouts
- WebSocket broadcast to all clients — filter SSE delivery by list subscription
**Research flag:** Standard patterns; no research phase needed.
---
### Phase 5: Web Push Notifications
**Rationale:** Push is built last because it depends on events being cached (Phase 2), users having stable identity + push_subscriptions table (Phase 1), and the service worker being in place (Phase 3).
**Delivers:** VAPID key pair; push_subscriptions table; push permission request UX (inside tap handler only); reminder scheduler (events within 15-min window); event-change and list-change push alerts; subscription health-check on app open; 410/404 cleanup; Declarative Web Push compatible payload format.
**Addresses:**
- Event reminders
- List-change notifications
- iOS push reliability
**Pitfalls to avoid:**
- `showNotification()` without `event.waitUntil()` — three silent pushes = permanent subscription revocation on iOS
- Fetch-before-display push handler — payload must be self-contained (title, body, URL); no API call inside push event handler
- Dead subscription accumulation — handle 410 Gone immediately; delete record
- EU DMA check — one-time: confirm both Apple IDs are non-EU; document result (expected: Canada, not affected)
- Declarative Web Push payload format — define the JSON schema before implementing server-side push sender
**Research flag:** Standard patterns for VAPID; iOS-specific behavior is well-documented in PITFALLS.md. No research phase needed, but the `waitUntil` pattern and subscription health-check must be in the initial implementation.
---
### Phase Ordering Rationale
- Phase 1 (auth + broker spike) is an unconditional prerequisite — nothing works without auth, and the personal-calendar ACL spike is a binary gate for the product's primary differentiator
- Phase 2 (calendar display) precedes Phase 3 (write-back) — validates broker correctness before adding write complexity
- Phase 4 (lists) could parallel-track with Phase 3 — no calendar dependency; serialized here to reduce WIP
- Phase 5 (push) is last by dependency: requires SW (Phase 3), event cache (Phase 2), stable user identity (Phase 1)
- The personal-calendar overlay feature is gated on the Phase 1 ACL spike result. If the spike confirms sharing works, overlay ships in Phase 2. If the wife's calendar is on iCloud, the overlay is cut from v1 scope with shared-family-calendar-only as the fallback.
### Research Flags
**Phases needing deeper research during planning:**
- **Phase 1 (CalDAV broker spike):** Fastmail personal-calendar CalDAV ACL sharing behavior — must be confirmed with a real account before Phase 2 calendar UI is built. Highest-risk item in the project.
- **Phase 1 (infra spike):** Pangolin SSE/WebSocket pass-through — known issue #1034; confirm transport choice and configure timeouts before real-time sync is built.
**Phases with standard patterns (skip research phase):**
- **Phase 2 (calendar display):** Calendar grid layout, CALDAV:expand, color coding — well-documented patterns.
- **Phase 3 (write-back + PWA):** CalDAV PUT mechanics, vite-plugin-pwa injectManifest — well-documented.
- **Phase 4 (lists + SSE):** REST CRUD + SSE + Redis pub/sub — standard patterns.
- **Phase 5 (push):** VAPID, web-push npm, iOS-specific patterns — well-documented in community sources; implementation-sensitive but not research-sensitive.
## Confidence Assessment
| Area | Confidence | Notes |
|------|------------|-------|
| Stack | HIGH | Strong library convergence across all four research threads; official sources for Fastmail CalDAV-only decision |
| Features | HIGH | Table stakes and anti-features well-evidenced across 5+ competing products; differentiators confirmed against PROJECT.md |
| Architecture | MEDIUM | Broker-cache pattern well-established; personal-calendar ACL behavior is the LOW-confidence gap |
| Pitfalls | MEDIUM-HIGH | CalDAV/RRULE/iOS push pitfalls well-documented in community; Fastmail-specific rate limits undocumented |
**Overall confidence:** MEDIUM-HIGH
### Gaps to Address
- **Personal calendar CalDAV ACL (critical):** Whether the broker token discovers the wife's personal calendar after Fastmail share+accept is unconfirmed. Must be spiked in Phase 1. If the wife uses iCloud Calendar as primary, personal overlay is impossible via CalDAV broker. Handle during Phase 1 planning: design the spike, define success criteria, define go/no-go decision point.
- **Wife's calendar location:** PROJECT.md assumes personal calendars are Fastmail-hosted. If the wife's primary calendar is iCloud, the unified view's differentiator is cut from v1. Confirm before Phase 2.
- **Fastmail CalDAV rate limits:** Not documented. 5-min ctag polling (one lightweight PROPFIND per calendar per tick) is conservative. Monitor for HTTP 429 in production; implement exponential backoff.
- **Declarative Web Push payload format:** `web-push` npm does not natively generate the `"web_push": 8030` JSON schema. VAPID remains the transport layer; the payload JSON must be hand-crafted to satisfy Declarative Web Push for iOS 18.4+. Define the payload schema in Phase 5 planning before implementing the server-side push sender.
## Sources
### Primary (HIGH confidence)
- [Fastmail API Documentation](https://www.fastmail.com/dev/) — CalDAV-only for calendars; JMAP calendar pending RFC 8984 finalization
- [Fastmail App Passwords](https://www.fastmail.help/hc/en-us/articles/360058752854-App-passwords) — App password scope covers CalDAV; OAuth not applicable
- [Fastmail Calendar Sharing](https://www.fastmail.help/hc/en-us/articles/1500000279781-Sharing-calendars-with-other-users) — ACL tiers confirmed; cross-account token access undocumented
- [Authelia OIDC Clients Configuration](https://www.authelia.com/configuration/identity-providers/openid-connect/clients/) — PKCE, groups claim, breaking changes
- [WebKit: Meet Declarative Web Push](https://webkit.org/blog/16535/meet-declarative-web-push/) — Safari 18.4+, ITP resilience
- [CalDAV ctag Extension](https://github.com/apple/ccs-calendarserver/blob/master/doc/Extensions/caldav-ctag.txt) — ctag-based polling
### Secondary (MEDIUM confidence)
- [MagicBell: PWA iOS Limitations 2026](https://www.magicbell.com/blog/pwa-ios-limitations-safari-support-complete-guide) — iOS 16.4+, Home Screen required, EU DMA
- [Sabre/DAV: Building a CalDAV client](https://sabre.io/dav/building-a-caldav-client/) — ETag, sync-token, VTIMEZONE pitfalls
- [Pangolin WebSocket issue #1034](https://github.com/fosrl/pangolin/issues/1034) — Known WebSocket upgrade failure through tunnel
- [Using Fastmail with CalDAV libraries](https://utf9k.net/blog/fastmail-caldav/) — Principal URL format, Cyrus IMAP internals
- tsdav, ical.js, rrule, web-push, @hono/oidc-auth npm packages — versions and compatibility confirmed
### Tertiary (LOW confidence)
- [Fastmail: Shared Calendaring Improvements](https://www.fastmail.com/blog/shared-calendar-improvements/) — CalDAV ACL standards alignment; cross-account broker token discovery unconfirmed
- [iOS push subscriptions terminated after 3 notifications](https://dev.to/progressier/how-to-fix-ios-push-subscriptions-being-terminated-after-3-notifications-39a7) — `event.waitUntil()` requirement; community-sourced
- [Home Assistant CalDAV all-day events UTC issue](https://github.com/home-assistant/core/issues/25814) — DATE vs DATETIME bug in production; real-world evidence
---
*Research completed: 2026-06-03*
*Ready for roadmap: yes*