luckberg/familysync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: generate project documentation

Browse Source
This commit is contained in:
Lucas Berger
2026-06-10 18:17:51 -04:00
parent 581b31916b
commit a9c3304c4e
10 changed files with 2031 additions and 279 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment.md
+172 -279
View File
@@ -1,342 +1,235 @@
# FamilySync — Deployment & Live-Verification Runbook
<!-- generated-by: gsd-doc-writer -->
# FamilySync — Deployment Guide
This document is the operator runbook for getting FamilySync running behind Authelia (OIDC)
and Pangolin/Newt (public tunnel), and for executing the **Phase 1 Gate 2** live-verification
items (`01-HUMAN-UAT.md`). It covers two deployment modes:
- **Mode A — Local test rig** (recommended for Gate 2): familysync + a Newt connector run on
your dev box, routed through your existing Pangolin under a *test* subdomain. Validates the
real external topology (HTTPS, Pangolin SSE pass-through, Authelia OIDC) **without** deploying
to Unraid and without touching the production stack.
- **Mode B — Unraid production**: the real household deployment. Identical app + config; only
the host and the Newt site differ.
> The behaviours Gate 2 is checking — Authelia OIDC redirect/session, and SSE survival through
> the tunnel — live in **Authelia** and **Pangolin/Newt**, not in *where* the origin container
> runs. So Mode A is a faithful test of both. Reserve Unraid (Mode B) for go-live.
Self-hosted Docker deployment on Unraid behind Authelia OIDC and a Pangolin/Newt outbound tunnel. The API serves the compiled React PWA as static files on a single port (3000), so only one route needs to be exposed through the tunnel.
---
## Topology
## Deployment Targets
```mermaid
flowchart LR
subgraph Public
User[Browser / iOS PWA]
Pangolin[Pangolin edge<br/>public HTTPS + WAF]
end
subgraph Private[Private network - no inbound ports]
Newt[Newt connector<br/>outbound tunnel]
API[familysync api<br/>Hono :3000]
DB[(MariaDB)]
Redis[(Redis - Phase 4)]
end
Authelia[Authelia OIDC<br/>auth.DOMAIN]
| Target | Config file |
|--------|-------------|
| Docker Compose (production) | `docker-compose.yml` |
| Docker Compose (dev override) | `docker-compose.dev.yml` |
| Container image | `apps/api/Dockerfile` (multi-stage, built from repo root) |
User -->|https://familysync.DOMAIN| Pangolin
Pangolin -->|tunnel| Newt
Newt --> API
API --> DB
API -.Phase 4.-> Redis
User -->|OIDC redirect| Authelia
API -->|token exchange / userinfo| Authelia
```
The production compose file brings up three services:
Key property: **Newt dials outbound to Pangolin** — there are no open inbound ports on the
private network (honours the project networking constraint). This is true for both modes.
| Service | Image | Purpose |
|---------|-------|---------|
| `api` | Built from `apps/api/Dockerfile` target `production` | Hono API + compiled React PWA, listens on port 3000 |
| `mariadb` | `mariadb:11` | Persistent MariaDB database |
| `redis` | `redis:7-alpine` | Present for live list sync (pub/sub); unused until Phase 4 |
---
## Prerequisites (both modes)
## Prerequisites
- A Pangolin instance you control, with a wildcard or per-host cert for `*.DOMAIN`.
- Authelia already deployed and reachable at `https://auth.DOMAIN` (project constraint).
- The familysync image builds: `docker compose build` (see repo `docker-compose.yml`).
- A Fastmail app password per member (scope "Mail, Contacts & Calendars") — see
`CAL-08-DECISION.md`. **Never commit it; it lives in a gitignored `.env`/`.env.spike`.**
### ⚠️ Same-parent-domain requirement (Pitfall 1)
FamilySync **must** be served under the same parent domain as Authelia so the session cookie is
same-site. e.g. Authelia at `auth.DOMAIN` and the app at `familysync.DOMAIN` (Mode B) or
`familysync-dev.DOMAIN` (Mode A). A different apex domain will break the OIDC session cookie.
- Docker and Docker Compose available on the Unraid host.
- Authelia already deployed with a FamilySync OIDC client registered (see [Register the OIDC Client](#register-the-oidc-client)).
- Pangolin/Newt tunnel configured to route an external HTTPS hostname to the Docker host on port 3000 (see [Pangolin / Newt Tunnel](#pangolin--newt-tunnel)).
- A `.env` file at the repo root with all required secrets (see [Environment Setup](#environment-setup)).
---
## Step 1 — Register the OIDC client in Authelia
## Register the OIDC Client
Authelia client registration is **additive** — adding a new `client_id` does not affect existing
clients, and is trivially reversible. For Mode A use a distinct id + redirect so it never collides
with the eventual production client.
Generate a hashed client secret:
```bash
authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72
# Record BOTH the plaintext (for the app's OIDC_CLIENT_SECRET) and the hash (for Authelia).
```
Add to Authelia `configuration.yml` under `identity_providers.oidc.clients`:
Add the following client block to your Authelia `configuration.yml` under `identity_providers.oidc.clients`:
```yaml
identity_providers:
oidc:
clients:
- client_id: 'familysync' # Mode A: 'familysync-dev'
client_name: 'FamilySync'
client_secret: '$pbkdf2-sha512$...' # the HASH from the command above
public: false
authorization_policy: 'one_factor'
redirect_uris:
- 'https://familysync.DOMAIN/callback' # Mode A: https://familysync-dev.DOMAIN/callback
scopes: [openid, profile, email]
response_types: [code]
grant_types: [authorization_code, refresh_token]
token_endpoint_auth_method: client_secret_basic
require_pkce: true
pkce_challenge_method: S256
- client_id: familysync
client_name: FamilySync
client_secret: '<pbkdf2-hash-of-your-plaintext-secret>'
public: false
authorization_policy: one_factor
redirect_uris:
- https://familysync.DOMAIN/callback # replace DOMAIN with your actual domain
scopes:
- openid
- profile
- email
- offline_access
response_types:
- code
grant_types:
- authorization_code
- refresh_token
require_pkce: true
pkce_challenge_method: S256
token_endpoint_auth_method: client_secret_basic
userinfo_signed_response_alg: none
```
Reload Authelia (`docker restart authelia` or its reload mechanism). These match the locked
auth params in `CLAUDE.md` (code flow + PKCE S256 + client_secret_basic).
These parameters are fixed — do not change `response_types`, `grant_types`, `require_pkce`, `pkce_challenge_method`, or `token_endpoint_auth_method`.
To generate the pbkdf2 hash from your chosen plaintext secret:
```bash
# Authelia CLI — run on the host where Authelia is installed
authelia crypto hash generate pbkdf2 --variant sha512
```
<!-- VERIFY: Authelia CLI command syntax may vary by version; verify against your installed Authelia release -->
Store the **plaintext** secret in `.env` as `OIDC_CLIENT_SECRET`. Never use the hash in `.env`.
---
## Step 2 — App environment (`.env`)
## Environment Setup
Copy `.env.example` → `.env` and fill in. Generate secrets as noted:
Copy `.env.example` to `.env` at the repo root and fill in every value. The file is gitignored and must never be committed.
```bash
# Session cookie signing secret for @hono/oidc-auth
OIDC_AUTH_SECRET=$(openssl rand -base64 32)
# Broker app-password encryption key (32 bytes hex)
APP_PASSWORD_ENCRYPTION_KEY=$(node -e "console.log(require('crypto').randomBytes(32).toString('hex'))")
```
Minimum production `.env`:
```dotenv
# Database
DB_HOST=mariadb
DB_PORT=3306
DB_USER=familysync
DB_PASSWORD=<strong>
DB_NAME=familysync
DB_ROOT_PASSWORD=<strong>
DB_PASSWORD=<strong-password>
DB_ROOT_PASSWORD=<strong-root-password>
# OIDC (Authelia)
OIDC_AUTH_SECRET=<openssl rand -base64 32>
# OIDC
OIDC_AUTH_SECRET=<run: openssl rand -base64 32>
OIDC_ISSUER=https://auth.DOMAIN
OIDC_CLIENT_ID=familysync # or familysync-dev (Mode A)
OIDC_CLIENT_SECRET=<plaintext secret matching the Authelia hash>
OIDC_CLIENT_SECRET=<plaintext-secret-matching-authelia-hash>
OIDC_REDIRECT_URI=https://familysync.DOMAIN/callback
# MANDATORY behind a tunnel — without it @hono/oidc-auth builds redirect_uri from the
# internal container hostname, which will not match the registered URI.
OIDC_AUTH_EXTERNAL_URL=https://familysync.DOMAIN
# Broker
APP_PASSWORD_ENCRYPTION_KEY=<64-hex>
# Broker (Fastmail app-password encryption)
APP_PASSWORD_ENCRYPTION_KEY=<run: node -e "console.log(require('crypto').randomBytes(32).toString('hex'))">
# Web Push (VAPID) — optional but required for push notifications
VAPID_PUBLIC_KEY=<base64-public-key>
VAPID_PRIVATE_KEY=<base64-private-key>
VAPID_SUBJECT=mailto:admin@example.com
```
`OIDC_CLIENT_ID` defaults to `familysync` and does not need to be set unless you registered a different ID in Authelia.
`OIDC_SCOPES` defaults to `openid profile email offline_access`. Do not add scopes that are not configured on the Authelia client.
**`DEV_AUTH_BYPASS` must NOT appear in the production `.env` or `docker-compose.yml`.** The API enforces this in code: when `NODE_ENV=production` the bypass is unconditionally disabled regardless of other variables, but omitting it entirely is the correct posture.
Generate VAPID keys:
```bash
npx web-push generate-vapid-keys --json
```
See [docs/CONFIGURATION.md](CONFIGURATION.md) for the full variable reference including optional variables and defaults.
---
## Step 3 — Apply the database schema
## Apply Database Migrations
The image does not auto-migrate. Bring up MariaDB and apply the committed migrations once:
The production image does **not** auto-migrate on startup. Migrations must be applied manually before the first container start and again after any schema change.
> **WARNING — do NOT use the `push` subcommand of drizzle-kit on this MariaDB.**
> The `mysql` dialect misreads MariaDB 11.x metadata and schedules a false truncate/recreate
> that **wipes data**. The `push` workflow has been removed from the project scripts for this
> reason. Always use the committed-migration path: `db:generate` to author a new migration
> (diffs `schema.ts` against committed JSON snapshots, never the live DB), `db:migrate` to apply it.
Migrations are run with `drizzle-kit`, which is a **devDependency**. The production image is built with `pnpm install --prod` (see `apps/api/Dockerfile`), so `drizzle-kit` is **not** present inside the running `api` container — you cannot migrate by exec-ing into it. Instead, run migrations from a host that has the full (dev) dependencies and can reach MariaDB.
The production `docker-compose.yml` does not expose the MariaDB port externally, so bring the database up with the dev compose override (which binds port 3306), apply the migrations from the host, then start the rest of the stack:
```bash
docker compose up -d mariadb
# from the repo root, host-side (dev override exposes 3306):
# 1. Bring up only MariaDB with the host port exposed
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d mariadb
# 2. Wait for it to be healthy
docker compose ps
# 3. Apply migrations from the host (requires dev deps installed: `pnpm install`)
DB_HOST=127.0.0.1 DB_PORT=3306 DB_USER=familysync DB_NAME=familysync DB_PASSWORD=<value> \
pnpm --filter @familysync/api exec drizzle-kit migrate
# verify: SHOW TABLES; -> users, member_credentials, calendars, calendar_events
pnpm --filter @familysync/api db:migrate
# 4. Start the full stack
docker compose up -d
```
To author a future schema change: edit `apps/api/src/db/schema.ts`, run
`pnpm --filter @familysync/api run db:generate` (diffs schema against committed snapshots — no DB
connection needed), commit the generated SQL, then apply with `db:migrate`.
**Never use `drizzle-kit push` against this MariaDB.** The `mysql` dialect mis-reads MariaDB 11.x schema metadata and schedules false destructive operations (table truncation). Always use `db:generate` + `db:migrate`.
---
## Step 4 — Pangolin route + Newt connector
## Pangolin / Newt Tunnel
In Pangolin, create a **resource/route** for the hostname:
The app is exposed to the public internet via an outbound Pangolin/Newt tunnel — no inbound ports are opened on the Unraid host.
- Host: `familysync.DOMAIN` (Mode A: `familysync-dev.DOMAIN`)
- Upstream: the Newt connector → `http://<api-host>:3000`
- Auth: leave Pangolin's own auth **off** for this route — FamilySync does its own Authelia OIDC
at the app layer. (Do not double-gate.)
<!-- VERIFY: Pangolin/Newt route configuration steps depend on your Pangolin dashboard and site-specific settings -->
### ⚠️ SSE idle timeout (Phase 4 dependency, issue #1034)
Configure the Pangolin route to forward HTTPS traffic for `https://familysync.DOMAIN` to `http://<docker-host-ip>:3000`. The `api` service in `docker-compose.yml` publishes port 3000 on the host:
FamilySync uses Server-Sent Events for live list sync (Phase 4). Long-lived SSE streams can be
killed by a proxy idle timeout. In the Pangolin route config, ensure response buffering is
**off** and the idle/read timeout is **>= 120s** (ideally higher). The Gate 2 SSE smoke test
below is what confirms this end to end — **it must pass before Phase 4 is built.**
```yaml
ports:
- "3000:3000"
```
### Newt connector
Ensure `OIDC_AUTH_EXTERNAL_URL` and `OIDC_REDIRECT_URI` in `.env` match the public hostname Pangolin exposes. Without `OIDC_AUTH_EXTERNAL_URL`, the OIDC middleware builds the callback URI from the internal container hostname, which will not match the URI registered in Authelia and will break the login flow.
- **Mode A (local):** run Newt on your dev box pointing at your Pangolin site token. It dials out;
no local ports are exposed. `familysync` (api) listens on `:3000` reachable by Newt.
- **Mode B (Unraid):** run the Newt container in the same Unraid stack (see Step 6).
---
## Build and Start
```bash
# Newt connector (example — use the site token Pangolin issues for this site)
docker run -d --name newt --restart unless-stopped \
-e PANGOLIN_ENDPOINT=https://pangolin.DOMAIN \
-e NEWT_ID=<site-id> -e NEWT_SECRET=<site-secret> \
fosrl/newt:latest
# From the repo root — builds both the API and the React PWA into one image
docker compose build
# Start all services
docker compose up -d
```
The Dockerfile uses a multi-stage build:
1. `builder` — compiles the TypeScript API (`pnpm --filter @familysync/api build`).
2. `pwa-builder` — builds the React PWA with Vite (`pnpm --filter @familysync/pwa build`).
3. `production` — installs production-only dependencies, copies the compiled API and the built PWA into `./public`. The API serves the PWA at `/` via `serveStatic`.
Both `builder` and `pwa-builder` stages run in parallel under BuildKit.
Rebuild after any source change:
```bash
docker compose build api && docker compose up -d api
```
The `api` service waits for the `mariadb` healthcheck to pass before starting (`depends_on: condition: service_healthy`).
---
## Health Check
The `/health` endpoint is unauthenticated and confirms a live database connection:
```bash
curl https://familysync.DOMAIN/health
# {"ok":true,"db":"up"}
```
A `503` response (`{"ok":false,"db":"down"}`) means the API cannot reach MariaDB. Check `docker compose logs api` and `docker compose logs mariadb`.
---
## Rollback
There is no automated rollback pipeline. To revert to a previous build:
1. Identify the prior working Git commit.
2. Stop the API: `docker compose stop api`.
3. Rebuild from the target commit: `git checkout <commit> && docker compose build api`.
4. Start: `docker compose up -d api`.
5. If the rollback crosses a schema migration boundary, restore the MariaDB volume from a backup — schema downgrades are not supported by Drizzle Kit's migrate command.
Take a MariaDB dump before every deployment that includes a migration:
```bash
docker compose exec mariadb mariadb-dump -u root -p familysync > backup-$(date +%Y%m%d).sql
```
---
## Step 5 — Bring up the app
## Monitoring
No monitoring agent is configured in the repository. The `/health` endpoint is available for uptime monitoring tools.
Application logs are written to stdout/stderr and captured by Docker:
```bash
docker compose up -d --build
curl -s http://localhost:3000/health # local sanity: {"ok":true,"db":"up"}
docker compose logs -f api
docker compose logs -f mariadb
```
Then proceed to Gate 2 verification through the public URL.
---
## Step 6 — Unraid production (Mode B only)
1. Copy the repo (or just `docker-compose.yml`, `apps/api/Dockerfile`, built image) to Unraid.
2. Create the `.env` on the Unraid host (do **not** commit it; store via Unraid's secrets/template).
3. Add the `newt` service to the production compose (or run as a separate Unraid container) bound
to the production Pangolin site.
4. Use a named Docker volume for `mariadb_data` on the array (not a throwaway volume).
5. `docker compose up -d --build`, then `drizzle-kit migrate` once (Step 3) against the prod DB.
6. Register the **production** Authelia client (`client_id: familysync`, prod redirect URI) if you
used `familysync-dev` for Mode A.
Differences from Mode A are limited to: host, Newt site token, volume location, and the OIDC
client id/redirect. The app code and `docker-compose.yml` are identical.
---
## Gate 2 — Live verification checklist (`01-HUMAN-UAT.md`)
Run from an **external** network (phone on cellular is ideal for a true external path).
| # | Item | Pass condition |
|---|------|----------------|
| 1 | **AUTH-01** login | `https://familysync.DOMAIN` → redirects to Authelia → after login, shell shows name + color + one cached event |
| 2 | **AUTH-02** session | Fully close + reopen browser → no re-login |
| 3 | **AUTH-03** colors | Wife logs in on iPhone → distinct, stable color |
| 4 | **iOS PWA** (pairs with Phase 3) | Add-to-Home-Screen, launch standalone, login completes in standalone mode (watch for redirect breaking out of standalone) |
| 5 | **SSE smoke** (gate before Phase 4) | Hold the stream open 5+ min without it being cut: |
```bash
# get the session cookie from the browser after logging in (DevTools → Application → Cookies)
curl -N -H "Cookie: oidc-auth=<value>" https://familysync.DOMAIN/api/sse/heartbeat
# expect a `heartbeat` event ~every 10s for 5+ minutes
```
- **SSE PASS** → SSE transport confirmed for Phase 4.
- **SSE FAIL** (stream cut early) → adjust Pangolin idle-timeout/buffering; if still failing, record
as a Phase 4 constraint and plan a reconnect/fallback strategy.
Record results in `.planning/phases/01-foundation-broker-spike/01-HUMAN-UAT.md`.
---
## Dev-auth bypass (Phase 2+ local development)
FamilySync builds Phase 2 and Phase 3 features behind a dev-auth bypass so live Authelia is not
required during development (D-14). The bypass injects a fixed dev user into the request context
and short-circuits the OIDC guard.
**Activation (local dev only):**
```bash
# In your local .env:
DEV_AUTH_BYPASS=true
NODE_ENV=development # or test, or any value other than 'production'
```
**Hard production guard:**
The bypass middleware's FIRST conditional is `process.env.NODE_ENV === 'production'`. If this is
true, the middleware returns a no-op passthrough regardless of any other env var. This means:
- Even if `DEV_AUTH_BYPASS=true` is accidentally present in a production container, it has zero
effect. The OIDC guard fires normally.
- The hard guard is checked before `DEV_AUTH_BYPASS` is read — there is no code path where
production + bypass = unauthenticated access.
**Production Docker Compose prohibition:**
The production `docker-compose.yml` MUST NOT include `DEV_AUTH_BYPASS` in the environment block.
The `.env.example` entry for `DEV_AUTH_BYPASS` is commented out by default as a reminder.
**What the bypass does:**
Sets `c.set('user', DEV_USER)` in the Hono context before `oidcAuthMiddleware` runs. Routes that
read `c.get('user')` receive a fixed dev user `{ id: 1, displayName: 'Dev User', color: '#4A90D9' }`.
Routes that call `getAuth(c)` from `@hono/oidc-auth` will still return null (no OIDC cookie is
present) — those routes must be updated to prefer `c.get('user')` when building Phase 2+.
### Running locally (host-side, no Docker)
Use this when you want to run the API and PWA directly on the host (no `docker compose up` for the
app containers), e.g. during Phase 2+ feature development with the dev-auth bypass active.
**Prerequisite: dev MariaDB must be running with the host port exposed.**
Use the dev compose override from Step 3 — it binds MariaDB to `localhost:3306`:
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d mariadb
```
**Why a plain `pnpm --filter @familysync/api dev` is not enough:**
The API `dev` script is `node --watch dist/index.js`. It does **not** auto-load `.env` — there is
no `dotenv` call and no `--env-file` flag in the script. Without any env vars, `db/client.ts`
defaults `DB_HOST` to `'localhost'`, which works for a host-side run. However, once you source the
root `.env` to pick up OIDC secrets and other variables, the problem surfaces: root `.env` sets
`DB_HOST=mariadb` (the Docker service name, only resolvable inside the Docker network). On the
host, `mariadb` does not resolve, so the DB connection fails.
The fix is to source `.env` for all the other variables and then immediately override `DB_HOST` back
to `localhost`.
**Run the API and PWA:**
Open two terminals from the repo root.
Terminal 1 — API:
```bash
# Build first (the dev script runs the compiled output, not ts-node)
pnpm --filter @familysync/api build
# Source root .env, then override DB_HOST and activate the bypass
set -a; source .env; set +a && DEV_AUTH_BYPASS=true DB_HOST=localhost pnpm --filter @familysync/api dev
```
Terminal 2 — PWA:
```bash
pnpm --filter @familysync/pwa dev
```
The `set -a; source .env; set +a` idiom exports every variable from the root `.env` into the shell
environment. The `DEV_AUTH_BYPASS=true DB_HOST=localhost` prefix on the same command line then
overrides those two specific vars for the `pnpm` child process — `DB_HOST=localhost` wins over the
`DB_HOST=mariadb` that was exported from `.env`.
**Why `--env-file` is not baked into the dev script:**
If `--env-file .env` were added to the API `dev` script, it would load `DB_HOST=mariadb`
automatically on every `pnpm dev` invocation. That value only works inside the Docker network; on
the host it resolves to nothing and the DB connection fails. Keeping `.env` loading out of the
script is intentional — the developer sources it manually and overrides `DB_HOST` as shown above.
<!-- VERIFY: No Sentry, Datadog, or OpenTelemetry dependency is present in package.json — confirm whether any external monitoring is wired at the infrastructure level -->