docs: capture D-14/D-15 (Gate 2 deferral + local-Newt test rig) + deployment runbook
- docs/deployment.md: operator runbook — Mode A (local Newt test rig) vs Mode B (Unraid prod), Authelia client block, env reference, Pangolin SSE idle-timeout note, Gate 2 checklist. - ROADMAP: Phase 1 verification-status note (Gate 1 done, Gate 2 deferred); Phase 3 gains live AUTH + iOS standalone-PWA criterion; Phase 4 gains hard SSE-smoke entry gate (#1034). - PROJECT.md: D-14 (Gate 2 deferral split) + D-15 (local Newt rig); CAL-08 marked validated. - STATE: decisions/blockers updated to reflect deferral and dev-auth-bypass approach.
This commit is contained in:
Lucas Berger
@@ -75,7 +75,9 @@ The household can see and co-edit one color-coded family calendar (shared + each
|
||||
| React PWA only, no React Native / App Store | App-like UX with one codebase for all surfaces; lowest onboarding friction (one URL) | — Pending |
|
||||
| Authelia OIDC for all members | True SSO consistent with existing infra; no separate calendar-credential problem | — Pending |
|
||||
| Web Push notifications in v1 (not deferred) | Family alerts judged essential day one; build push into the foundation | — Pending |
|
||||
| Personal-calendar overlay in v1 (not shared-only) | Unified view of everyone's schedules is the Skylight magic worth shipping early | — Pending |
|
||||
| Personal-calendar overlay in v1 (not shared-only) | Unified view of everyone's schedules is the Skylight magic worth shipping early | — Validated (CAL-08 GO, Phase 1): per-member app password reaches all of an account's calendars; no cross-account ACL needed |
|
||||
| **D-14:** Defer Phase 1 Gate 2 (live Authelia/Pangolin verification). SSE-over-Pangolin smoke = hard gate before Phase 4; live AUTH smoke incl. iOS standalone-PWA folded into Phase 3; full 2-member prod login verified there. Phases 2–3 develop behind a documented dev-auth bypass. | Gate 2 needs operator infra (Authelia config + tunnel) + docs that didn't exist; deferring unblocks Phase 2/3 code without rework risk, since the broker data path (CAL-01/CAL-08) is already proven live. SSE must still be verified before Phase 4 to avoid building live-sync on an unverified transport (#1034). | Tracked: `01-HUMAN-UAT.md`, `docs/deployment.md` |
|
||||
| **D-15:** Validate the real external topology via a **local Newt connector + test subdomain** through existing Pangolin (Mode A), not an Unraid deploy. Unraid (Mode B) reserved for go-live. | Authelia OIDC + SSE pass-through behaviour live in Authelia + Pangolin/Newt, not in where the origin runs — so a local Newt rig faithfully tests both, decoupling "does the topology work" from "is it in production." Newt dials outbound (no open ports). Only shared touch is an additive, reversible Authelia client. | — Pending (Gate 2) |
|
||||
|
||||
## Evolution
|
||||
|
||||
|
||||
@@ -35,6 +35,8 @@ Decimal phases appear between their surrounding integers in numeric order.
|
||||
4. The broker successfully fetches and caches at least one event from the shared Fastmail calendar via CalDAV PROPFIND/REPORT
|
||||
5. The personal-calendar ACL spike produces a documented go/no-go decision: either the broker token sees the wife's personal calendar after Fastmail share+accept, or the fallback strategy (shared-family-only or per-member app password) is chosen and recorded
|
||||
|
||||
**Verification status (D-14, 2026-06-04):** Code + **Gate 1** complete. Gate 1 = stack up (`/health` live), CAL-01 proven live (503 real events cached via REPORT), CAL-08 = **GO** (per-member app-password model, see `CAL-08-DECISION.md`). **Gate 2 deferred** — criteria 1/2/3 (live Authelia OIDC login over Pangolin, session persistence, distinct colors in a real browser) and the SSE-over-Pangolin smoke test require the operator's Authelia + Pangolin/Newt infra; tracked in `01-HUMAN-UAT.md` and `docs/deployment.md`. The live AUTH smoke (incl. iOS) is folded into **Phase 3**; the SSE smoke is a hard gate before **Phase 4**. Phases 2–3 develop behind a documented dev-auth bypass.
|
||||
|
||||
**Plans**: 4 plans
|
||||
Plans:
|
||||
|
||||
@@ -72,6 +74,7 @@ Plans:
|
||||
3. A member can delete an event and it disappears from all views on the next sync
|
||||
4. On Android, the app shows a browser install prompt and installs to the home screen; on iOS, the app shows a guided "Add to Home Screen" walkthrough with annotated screenshots that a non-technical user can follow independently
|
||||
5. The installed PWA opens full-screen without browser chrome on both iOS and Android
|
||||
6. **(Carried from Phase 1 Gate 2, D-14)** Live Authelia OIDC login works over the public Pangolin URL — including the **iOS standalone-PWA** flow: the wife can install to Home Screen and complete login without the redirect breaking out of standalone mode; sessions persist (AUTH-01/02) and members get distinct stable colors (AUTH-03). Verify per `docs/deployment.md` Gate 2 checklist; this is the first real external deploy (local Newt test rig is sufficient — Unraid prod is optional until go-live).
|
||||
|
||||
**Plans**: TBD
|
||||
**UI hint**: yes
|
||||
@@ -81,6 +84,7 @@ Plans:
|
||||
**Goal**: Both members can create and manage shared named lists with real-time co-edit sync — edits by one member appear for the other without any manual refresh
|
||||
**Mode:** mvp
|
||||
**Depends on**: Phase 1
|
||||
**⚠️ Entry gate (D-14):** The **SSE-over-Pangolin smoke test** (deferred from Phase 1 Gate 2, issue #1034) MUST pass before building live sync — hold `/api/sse/heartbeat` open 5+ min through the tunnel without it being cut (see `docs/deployment.md`). If it FAILS: fix Pangolin idle-timeout/buffering, or plan a reconnect/polling fallback into this phase before proceeding. Do not build the live-sync layer on an unverified transport.
|
||||
**Requirements**: LIST-01, LIST-02, LIST-03, LIST-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
|
||||
|
||||
+11
-8
@@ -60,10 +60,12 @@ Progress: [░░░░░░░░░░] 0%
|
||||
Decisions are logged in PROJECT.md Key Decisions table.
|
||||
Recent decisions affecting current work:
|
||||
|
||||
- Phase 1 gate: Personal-calendar CalDAV ACL must be spiked before calendar UI is built. Fallback is shared-family-only if spike fails.
|
||||
- CAL-08 RESOLVED → GO (Phase 1): per-member Fastmail app password reaches all of that account's calendars; no cross-account ACL needed. Unified view stands; no shared-only fallback. See CAL-08-DECISION.md.
|
||||
- D-14 (2026-06-04): Phase 1 Gate 2 (live Authelia/Pangolin) deferred. SSE-over-Pangolin smoke = hard gate before Phase 4; live AUTH smoke incl. iOS standalone-PWA folded into Phase 3. Phases 2–3 build behind a dev-auth bypass. Tracked in 01-HUMAN-UAT.md + docs/deployment.md.
|
||||
- D-15 (2026-06-04): Validate real topology via local Newt connector + test subdomain through Pangolin (Mode A), not an Unraid deploy; Unraid reserved for go-live.
|
||||
- CalDAV locked: Fastmail does not expose calendars over JMAP. CalDAV via tsdav is the only protocol. No reconsideration.
|
||||
- Identity: Use oidc_iss + oidc_sub as stable composite key. Never email.
|
||||
- Real-time transport: Prefer SSE over WebSocket (proxy-resilient through Pangolin). Verify Pangolin SSE pass-through in Phase 1 infra spike.
|
||||
- Real-time transport: Prefer SSE over WebSocket (proxy-resilient through Pangolin). SSE pass-through STILL UNVERIFIED — gate before Phase 4 (D-14).
|
||||
- Recurring events: Create + display only in v1 (CALDAV:expand on server side). Single-occurrence edit deferred to v1.x.
|
||||
|
||||
### Pending Todos
|
||||
@@ -72,9 +74,10 @@ None yet.
|
||||
|
||||
### Blockers/Concerns
|
||||
|
||||
- Phase 1: Personal-calendar CalDAV ACL behavior on Fastmail is LOW confidence (must spike). Failure degrades unified view to shared-family-only for v1.
|
||||
- Phase 1: Pangolin SSE/WebSocket pass-through is an open infra question (known issue #1034). Must smoke-test before Phase 4 real-time sync is built.
|
||||
- Phase 3: iOS install guide is load-bearing for the wife — she will never receive push notifications if she does not install the PWA.
|
||||
- ~~Phase 1: Personal-calendar CalDAV ACL~~ RESOLVED → CAL-08 GO (per-member app password; no cross-account ACL).
|
||||
- Phase 4 ENTRY GATE: Pangolin SSE pass-through (issue #1034) still unverified — deferred from Phase 1 Gate 2 (D-14). Must pass the 5-min SSE smoke (docs/deployment.md) before building live sync.
|
||||
- Phase 3: iOS standalone-PWA + Authelia login is load-bearing for the wife and is the first real external auth test (carried Gate 2 item, D-14). Also: iOS install guide is load-bearing — she gets no push notifications if she does not install the PWA.
|
||||
- Phase 2/3 dev: build behind a documented dev-auth bypass until Gate 2 deploy (D-14).
|
||||
- Phase 5: iOS push subscriptions silently revoked after 3 silent pushes. Subscription health-check and event.waitUntil() are mandatory from day one.
|
||||
|
||||
## Deferred Items
|
||||
@@ -89,6 +92,6 @@ None yet.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-06-04T11:55:07.995Z
|
||||
Stopped at: Phase 1 context gathered
|
||||
Resume file: .planning/phases/01-foundation-broker-spike/01-CONTEXT.md
|
||||
Last session: 2026-06-04
|
||||
Stopped at: Phase 1 code + Gate 1 complete (CAL-01/CAL-08 proven live). Gate 2 (live Authelia/Pangolin + SSE) deferred per D-14, tracked in 01-HUMAN-UAT.md + docs/deployment.md. Decisions D-14/D-15 logged. Ready to plan Phase 2.
|
||||
Resume file: .planning/phases/01-foundation-broker-spike/01-HUMAN-UAT.md
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
# FamilySync — Deployment & Live-Verification Runbook
|
||||
|
||||
This document is the operator runbook for getting FamilySync running behind Authelia (OIDC)
|
||||
and Pangolin/Newt (public tunnel), and for executing the **Phase 1 Gate 2** live-verification
|
||||
items (`01-HUMAN-UAT.md`). It covers two deployment modes:
|
||||
|
||||
- **Mode A — Local test rig** (recommended for Gate 2): familysync + a Newt connector run on
|
||||
your dev box, routed through your existing Pangolin under a *test* subdomain. Validates the
|
||||
real external topology (HTTPS, Pangolin SSE pass-through, Authelia OIDC) **without** deploying
|
||||
to Unraid and without touching the production stack.
|
||||
- **Mode B — Unraid production**: the real household deployment. Identical app + config; only
|
||||
the host and the Newt site differ.
|
||||
|
||||
> The behaviours Gate 2 is checking — Authelia OIDC redirect/session, and SSE survival through
|
||||
> the tunnel — live in **Authelia** and **Pangolin/Newt**, not in *where* the origin container
|
||||
> runs. So Mode A is a faithful test of both. Reserve Unraid (Mode B) for go-live.
|
||||
|
||||
---
|
||||
|
||||
## Topology
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Public
|
||||
User[Browser / iOS PWA]
|
||||
Pangolin[Pangolin edge<br/>public HTTPS + WAF]
|
||||
end
|
||||
subgraph Private[Private network - no inbound ports]
|
||||
Newt[Newt connector<br/>outbound tunnel]
|
||||
API[familysync api<br/>Hono :3000]
|
||||
DB[(MariaDB)]
|
||||
Redis[(Redis - Phase 4)]
|
||||
end
|
||||
Authelia[Authelia OIDC<br/>auth.DOMAIN]
|
||||
|
||||
User -->|https://familysync.DOMAIN| Pangolin
|
||||
Pangolin -->|tunnel| Newt
|
||||
Newt --> API
|
||||
API --> DB
|
||||
API -.Phase 4.-> Redis
|
||||
User -->|OIDC redirect| Authelia
|
||||
API -->|token exchange / userinfo| Authelia
|
||||
```
|
||||
|
||||
Key property: **Newt dials outbound to Pangolin** — there are no open inbound ports on the
|
||||
private network (honours the project networking constraint). This is true for both modes.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites (both modes)
|
||||
|
||||
- A Pangolin instance you control, with a wildcard or per-host cert for `*.DOMAIN`.
|
||||
- Authelia already deployed and reachable at `https://auth.DOMAIN` (project constraint).
|
||||
- The familysync image builds: `docker compose build` (see repo `docker-compose.yml`).
|
||||
- A Fastmail app password per member (scope "Mail, Contacts & Calendars") — see
|
||||
`CAL-08-DECISION.md`. **Never commit it; it lives in a gitignored `.env`/`.env.spike`.**
|
||||
|
||||
### ⚠️ Same-parent-domain requirement (Pitfall 1)
|
||||
|
||||
FamilySync **must** be served under the same parent domain as Authelia so the session cookie is
|
||||
same-site. e.g. Authelia at `auth.DOMAIN` and the app at `familysync.DOMAIN` (Mode B) or
|
||||
`familysync-dev.DOMAIN` (Mode A). A different apex domain will break the OIDC session cookie.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Register the OIDC client in Authelia
|
||||
|
||||
Authelia client registration is **additive** — adding a new `client_id` does not affect existing
|
||||
clients, and is trivially reversible. For Mode A use a distinct id + redirect so it never collides
|
||||
with the eventual production client.
|
||||
|
||||
Generate a hashed client secret:
|
||||
|
||||
```bash
|
||||
authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72
|
||||
# Record BOTH the plaintext (for the app's OIDC_CLIENT_SECRET) and the hash (for Authelia).
|
||||
```
|
||||
|
||||
Add to Authelia `configuration.yml` under `identity_providers.oidc.clients`:
|
||||
|
||||
```yaml
|
||||
identity_providers:
|
||||
oidc:
|
||||
clients:
|
||||
- client_id: 'familysync' # Mode A: 'familysync-dev'
|
||||
client_name: 'FamilySync'
|
||||
client_secret: '$pbkdf2-sha512$...' # the HASH from the command above
|
||||
public: false
|
||||
authorization_policy: 'one_factor'
|
||||
redirect_uris:
|
||||
- 'https://familysync.DOMAIN/callback' # Mode A: https://familysync-dev.DOMAIN/callback
|
||||
scopes: [openid, profile, email]
|
||||
response_types: [code]
|
||||
grant_types: [authorization_code, refresh_token]
|
||||
token_endpoint_auth_method: client_secret_basic
|
||||
require_pkce: true
|
||||
pkce_challenge_method: S256
|
||||
```
|
||||
|
||||
Reload Authelia (`docker restart authelia` or its reload mechanism). These match the locked
|
||||
auth params in `CLAUDE.md` (code flow + PKCE S256 + client_secret_basic).
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — App environment (`.env`)
|
||||
|
||||
Copy `.env.example` → `.env` and fill in. Generate secrets as noted:
|
||||
|
||||
```bash
|
||||
# Session cookie signing secret for @hono/oidc-auth
|
||||
OIDC_AUTH_SECRET=$(openssl rand -base64 32)
|
||||
# Broker app-password encryption key (32 bytes hex)
|
||||
APP_PASSWORD_ENCRYPTION_KEY=$(node -e "console.log(require('crypto').randomBytes(32).toString('hex'))")
|
||||
```
|
||||
|
||||
```dotenv
|
||||
# Database
|
||||
DB_HOST=mariadb
|
||||
DB_PORT=3306
|
||||
DB_USER=familysync
|
||||
DB_PASSWORD=<strong>
|
||||
DB_NAME=familysync
|
||||
DB_ROOT_PASSWORD=<strong>
|
||||
|
||||
# OIDC (Authelia)
|
||||
OIDC_AUTH_SECRET=<openssl rand -base64 32>
|
||||
OIDC_ISSUER=https://auth.DOMAIN
|
||||
OIDC_CLIENT_ID=familysync # or familysync-dev (Mode A)
|
||||
OIDC_CLIENT_SECRET=<plaintext secret matching the Authelia hash>
|
||||
OIDC_REDIRECT_URI=https://familysync.DOMAIN/callback
|
||||
# MANDATORY behind a tunnel — without it @hono/oidc-auth builds redirect_uri from the
|
||||
# internal container hostname, which will not match the registered URI.
|
||||
OIDC_AUTH_EXTERNAL_URL=https://familysync.DOMAIN
|
||||
|
||||
# Broker
|
||||
APP_PASSWORD_ENCRYPTION_KEY=<64-hex>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Apply the database schema
|
||||
|
||||
The image does not auto-migrate. Bring up MariaDB and push the Drizzle schema once:
|
||||
|
||||
```bash
|
||||
docker compose up -d mariadb
|
||||
# from the repo root, host-side (dev override exposes 3306):
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d mariadb
|
||||
DB_HOST=127.0.0.1 DB_PORT=3306 DB_USER=familysync DB_NAME=familysync DB_PASSWORD=<value> \
|
||||
pnpm --filter @familysync/api exec drizzle-kit push
|
||||
# verify: SHOW TABLES; -> users, member_credentials, calendars, calendar_events
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Pangolin route + Newt connector
|
||||
|
||||
In Pangolin, create a **resource/route** for the hostname:
|
||||
|
||||
- Host: `familysync.DOMAIN` (Mode A: `familysync-dev.DOMAIN`)
|
||||
- Upstream: the Newt connector → `http://<api-host>:3000`
|
||||
- Auth: leave Pangolin's own auth **off** for this route — FamilySync does its own Authelia OIDC
|
||||
at the app layer. (Do not double-gate.)
|
||||
|
||||
### ⚠️ SSE idle timeout (Phase 4 dependency, issue #1034)
|
||||
|
||||
FamilySync uses Server-Sent Events for live list sync (Phase 4). Long-lived SSE streams can be
|
||||
killed by a proxy idle timeout. In the Pangolin route config, ensure response buffering is
|
||||
**off** and the idle/read timeout is **>= 120s** (ideally higher). The Gate 2 SSE smoke test
|
||||
below is what confirms this end to end — **it must pass before Phase 4 is built.**
|
||||
|
||||
### Newt connector
|
||||
|
||||
- **Mode A (local):** run Newt on your dev box pointing at your Pangolin site token. It dials out;
|
||||
no local ports are exposed. `familysync` (api) listens on `:3000` reachable by Newt.
|
||||
- **Mode B (Unraid):** run the Newt container in the same Unraid stack (see Step 6).
|
||||
|
||||
```bash
|
||||
# Newt connector (example — use the site token Pangolin issues for this site)
|
||||
docker run -d --name newt --restart unless-stopped \
|
||||
-e PANGOLIN_ENDPOINT=https://pangolin.DOMAIN \
|
||||
-e NEWT_ID=<site-id> -e NEWT_SECRET=<site-secret> \
|
||||
fosrl/newt:latest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Bring up the app
|
||||
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
curl -s http://localhost:3000/health # local sanity: {"ok":true,"db":"up"}
|
||||
```
|
||||
|
||||
Then proceed to Gate 2 verification through the public URL.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Unraid production (Mode B only)
|
||||
|
||||
1. Copy the repo (or just `docker-compose.yml`, `apps/api/Dockerfile`, built image) to Unraid.
|
||||
2. Create the `.env` on the Unraid host (do **not** commit it; store via Unraid's secrets/template).
|
||||
3. Add the `newt` service to the production compose (or run as a separate Unraid container) bound
|
||||
to the production Pangolin site.
|
||||
4. Use a named Docker volume for `mariadb_data` on the array (not a throwaway volume).
|
||||
5. `docker compose up -d --build`, then `drizzle-kit push` once (Step 3) against the prod DB.
|
||||
6. Register the **production** Authelia client (`client_id: familysync`, prod redirect URI) if you
|
||||
used `familysync-dev` for Mode A.
|
||||
|
||||
Differences from Mode A are limited to: host, Newt site token, volume location, and the OIDC
|
||||
client id/redirect. The app code and `docker-compose.yml` are identical.
|
||||
|
||||
---
|
||||
|
||||
## Gate 2 — Live verification checklist (`01-HUMAN-UAT.md`)
|
||||
|
||||
Run from an **external** network (phone on cellular is ideal for a true external path).
|
||||
|
||||
| # | Item | Pass condition |
|
||||
|---|------|----------------|
|
||||
| 1 | **AUTH-01** login | `https://familysync.DOMAIN` → redirects to Authelia → after login, shell shows name + color + one cached event |
|
||||
| 2 | **AUTH-02** session | Fully close + reopen browser → no re-login |
|
||||
| 3 | **AUTH-03** colors | Wife logs in on iPhone → distinct, stable color |
|
||||
| 4 | **iOS PWA** (pairs with Phase 3) | Add-to-Home-Screen, launch standalone, login completes in standalone mode (watch for redirect breaking out of standalone) |
|
||||
| 5 | **SSE smoke** (gate before Phase 4) | Hold the stream open 5+ min without it being cut: |
|
||||
|
||||
```bash
|
||||
# get the session cookie from the browser after logging in (DevTools → Application → Cookies)
|
||||
curl -N -H "Cookie: oidc-auth=<value>" https://familysync.DOMAIN/api/sse/heartbeat
|
||||
# expect a `heartbeat` event ~every 10s for 5+ minutes
|
||||
```
|
||||
|
||||
- **SSE PASS** → SSE transport confirmed for Phase 4.
|
||||
- **SSE FAIL** (stream cut early) → adjust Pangolin idle-timeout/buffering; if still failing, record
|
||||
as a Phase 4 constraint and plan a reconnect/fallback strategy.
|
||||
|
||||
Record results in `.planning/phases/01-foundation-broker-spike/01-HUMAN-UAT.md`.
|
||||
Reference in New Issue
Block a user