From 7a089421dd715be65c263a29398d7f02cac0d68e Mon Sep 17 00:00:00 2001 From: Lucas Berger Date: Wed, 10 Jun 2026 22:24:52 -0400 Subject: [PATCH] docs(07): add validation strategy --- .../07-mobile-test-harness/07-VALIDATION.md | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 .planning/phases/07-mobile-test-harness/07-VALIDATION.md diff --git a/.planning/phases/07-mobile-test-harness/07-VALIDATION.md b/.planning/phases/07-mobile-test-harness/07-VALIDATION.md new file mode 100644 index 0000000..6e7cf67 --- /dev/null +++ b/.planning/phases/07-mobile-test-harness/07-VALIDATION.md @@ -0,0 +1,100 @@ +--- +phase: 07 +slug: mobile-test-harness +status: draft +nyquist_compliant: false +wave_0_complete: false +created: 2026-06-10 +--- + +# Phase 07 — Validation Strategy + +> Per-phase validation contract for feedback sampling during execution. +> This phase's deliverable IS the test infrastructure, so "validation" here means +> proving the harness itself detects real defects (see Harness Self-Validation). + +--- + +## Test Infrastructure + +| Property | Value | +|----------|-------| +| **Framework** | `@playwright/test` 1.60.0 (new dev dep in `apps/pwa`) | +| **Config file** | `apps/pwa/playwright.config.ts` (none today — Wave 0 creates it) | +| **Quick run command** | `pnpm --filter @familysync/pwa exec playwright test --project=pixel` | +| **Full suite command** | `pnpm --filter @familysync/pwa exec playwright test` | +| **CI invocation** | `CI=true pnpm --filter @familysync/pwa exec playwright test` | +| **Estimated runtime** | ~30–60s full suite (two profiles, host stack already up) | + +> Existing Vitest unit suite (`apps/pwa`, `*.test.ts`) remains the per-commit unit gate; this phase adds a **separate** E2E suite (`e2e/*.spec.ts`). The two must not share a glob — Wave 0 adds `exclude: ['e2e/**']` to `vitest.config.ts`. + +--- + +## Sampling Rate + +- **After every task commit:** Run `pnpm --filter @familysync/pwa exec playwright test --project=pixel` (Chromium-only, faster feedback) +- **After every plan wave:** Run `pnpm --filter @familysync/pwa exec playwright test` (both iPhone/WebKit + Pixel/Chromium profiles) +- **Before `/gsd-verify-work`:** Full suite green on **both** profiles +- **Max feedback latency:** ~60 seconds (full suite, host stack running) + +--- + +## Per-Task Verification Map + +> Task IDs resolve when PLAN.md files are written; rows below are keyed by requirement + target file so the planner can attach `` verify blocks. Every Phase-7 task must map to one of these or declare a Wave 0 dependency. + +| Plan/Wave | Requirement | Behavior verified | Test Type | Automated Command | File (Wave 0) | Status | +|-----------|-------------|-------------------|-----------|-------------------|---------------|--------| +| W0 | TEST-01/02 | `@playwright/test` installed; config matrix (iPhone WebKit + Pixel Chromium), `serviceWorkers: 'block'`, env baseURL | config | `pnpm --filter @familysync/pwa exec playwright test --list` | `apps/pwa/playwright.config.ts` | ⬜ pending | +| W0 | TEST-02 | global-setup polls `/health` then deterministically seeds (calendar id 10 `INSERT IGNORE` FK guard + list + items, reset-per-run) | infra | run produces seeded rows; spec reads populated views | `apps/pwa/e2e/global-setup.ts` | ⬜ pending | +| W1 | TEST-01 | Tap targets ≥ 44px (BottomTabBar, FAB, Retry, settings); no horizontal overflow on `/calendar` `/lists` | E2E | `pnpm --filter @familysync/pwa exec playwright test` | `apps/pwa/e2e/layout.spec.ts` | ⬜ pending | +| W1 | TEST-01 | Calendar populated + empty + error states (UI-SPEC Rules 4/5) | E2E | same | `apps/pwa/e2e/calendar.spec.ts` | ⬜ pending | +| W1 | TEST-01 | Lists populated + empty states (UI-SPEC Rules 4/5) | E2E | same | `apps/pwa/e2e/lists.spec.ts` | ⬜ pending | +| W1 | TEST-02 | Authenticated PWA reached via `DEV_AUTH_BYPASS` (no manual login, no OIDC mock); trace shows no SW-sourced responses | E2E | `CI=true pnpm --filter @familysync/pwa exec playwright test` | spec preconditions + trace | ⬜ pending | + +*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky* + +--- + +## Harness Self-Validation (the harness must prove it works) + +The deliverable is test infrastructure — a green suite alone does not prove it would catch a real defect. The phase is validated only when the assertions provably fail on injected defects: + +1. **Tap-target failure injection** — temporarily shrink a touch target (e.g. BottomTabBar `minHeight: 20px` via `page.addStyleTag`); the ≥44px assertion MUST fail. Restore → passes. Proves `boundingBox()` measures the rendered element, not the CSS source. +2. **Overflow failure injection** — add `body { width: 2000px }` via `page.addStyleTag` before the overflow assertion; it MUST fail. Remove → passes. Proves the `scrollWidth > clientWidth` check is live. +3. **SW-block verification** — with `trace: 'on-first-retry'`, inspect one trace and confirm **zero** responses sourced from `(ServiceWorker)` (D-02 / Pitfall 15). +4. **Auth-bypass verification (manual)** — with `DEV_AUTH_BYPASS` disabled the API redirects to Authelia and specs MUST fail on missing authed content; with it enabled they pass (D-01 / SC #2). + +--- + +## Wave 0 Requirements + +- [ ] `apps/pwa/playwright.config.ts` — project matrix (iPhone WebKit + Pixel Chromium), `globalSetup`, optional vite-only `webServer` with `reuseExistingServer: !process.env.CI`, env-driven `baseURL`, artifact/trace config +- [ ] `apps/pwa/e2e/global-setup.ts` — `/health` readiness poll + deterministic DB seed (calendar id 10 `INSERT IGNORE` guard + list + items via `mysql2`, reset-per-run) +- [ ] `apps/pwa/vitest.config.ts` — add `exclude: ['e2e/**']` to prevent the `*.spec.ts` glob collision +- [ ] `apps/pwa/package.json` — add `@playwright/test` devDependency + `test:e2e` script +- [ ] Browser install: `pnpm --filter @familysync/pwa exec playwright install --with-deps webkit chromium` +- [ ] `apps/pwa/tsconfig`/typecheck — ensure `e2e/**` passes the `tsc --noEmit` gate + +--- + +## Manual-Only Verifications + +| Behavior | Requirement | Why Manual | Test Instructions | +|----------|-------------|------------|-------------------| +| Auth-bypass off → Authelia redirect | TEST-02 / SC #2 | Requires flipping `DEV_AUTH_BYPASS` off on the dev stack (env change outside the spec) | Disable bypass, run suite, confirm specs fail on missing authed content; re-enable, confirm green | +| Real prod-service-worker behavior | (out of scope) | Harness blocks the SW by design (D-02); prod SW is a device gate | Human/device check — not automated here | +| iOS-Safari standalone-PWA (Home-Screen install, standalone OIDC redirect, iOS push) | (out of scope) | Cannot be driven by Playwright/WebKit emulation | Human/device gate per project CLAUDE.md exception | + +--- + +## Validation Sign-Off + +- [ ] All tasks have `` verify or a Wave 0 dependency +- [ ] Sampling continuity: no 3 consecutive tasks without automated verify +- [ ] Wave 0 covers all MISSING references (config, global-setup, browser install) +- [ ] No watch-mode flags in committed commands +- [ ] Feedback latency < 60s +- [ ] `nyquist_compliant: true` set in frontmatter (by planner once task map is complete) + +**Approval:** pending