docs(07): add validation strategy
This commit is contained in:
Lucas Berger
@@ -0,0 +1,100 @@
|
|||||||
|
---
|
||||||
|
phase: 07
|
||||||
|
slug: mobile-test-harness
|
||||||
|
status: draft
|
||||||
|
nyquist_compliant: false
|
||||||
|
wave_0_complete: false
|
||||||
|
created: 2026-06-10
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 07 — Validation Strategy
|
||||||
|
|
||||||
|
> Per-phase validation contract for feedback sampling during execution.
|
||||||
|
> This phase's deliverable IS the test infrastructure, so "validation" here means
|
||||||
|
> proving the harness itself detects real defects (see Harness Self-Validation).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Test Infrastructure
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|----------|-------|
|
||||||
|
| **Framework** | `@playwright/test` 1.60.0 (new dev dep in `apps/pwa`) |
|
||||||
|
| **Config file** | `apps/pwa/playwright.config.ts` (none today — Wave 0 creates it) |
|
||||||
|
| **Quick run command** | `pnpm --filter @familysync/pwa exec playwright test --project=pixel` |
|
||||||
|
| **Full suite command** | `pnpm --filter @familysync/pwa exec playwright test` |
|
||||||
|
| **CI invocation** | `CI=true pnpm --filter @familysync/pwa exec playwright test` |
|
||||||
|
| **Estimated runtime** | ~30–60s full suite (two profiles, host stack already up) |
|
||||||
|
|
||||||
|
> Existing Vitest unit suite (`apps/pwa`, `*.test.ts`) remains the per-commit unit gate; this phase adds a **separate** E2E suite (`e2e/*.spec.ts`). The two must not share a glob — Wave 0 adds `exclude: ['e2e/**']` to `vitest.config.ts`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sampling Rate
|
||||||
|
|
||||||
|
- **After every task commit:** Run `pnpm --filter @familysync/pwa exec playwright test --project=pixel` (Chromium-only, faster feedback)
|
||||||
|
- **After every plan wave:** Run `pnpm --filter @familysync/pwa exec playwright test` (both iPhone/WebKit + Pixel/Chromium profiles)
|
||||||
|
- **Before `/gsd-verify-work`:** Full suite green on **both** profiles
|
||||||
|
- **Max feedback latency:** ~60 seconds (full suite, host stack running)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Per-Task Verification Map
|
||||||
|
|
||||||
|
> Task IDs resolve when PLAN.md files are written; rows below are keyed by requirement + target file so the planner can attach `<automated>` verify blocks. Every Phase-7 task must map to one of these or declare a Wave 0 dependency.
|
||||||
|
|
||||||
|
| Plan/Wave | Requirement | Behavior verified | Test Type | Automated Command | File (Wave 0) | Status |
|
||||||
|
|-----------|-------------|-------------------|-----------|-------------------|---------------|--------|
|
||||||
|
| W0 | TEST-01/02 | `@playwright/test` installed; config matrix (iPhone WebKit + Pixel Chromium), `serviceWorkers: 'block'`, env baseURL | config | `pnpm --filter @familysync/pwa exec playwright test --list` | `apps/pwa/playwright.config.ts` | ⬜ pending |
|
||||||
|
| W0 | TEST-02 | global-setup polls `/health` then deterministically seeds (calendar id 10 `INSERT IGNORE` FK guard + list + items, reset-per-run) | infra | run produces seeded rows; spec reads populated views | `apps/pwa/e2e/global-setup.ts` | ⬜ pending |
|
||||||
|
| W1 | TEST-01 | Tap targets ≥ 44px (BottomTabBar, FAB, Retry, settings); no horizontal overflow on `/calendar` `/lists` | E2E | `pnpm --filter @familysync/pwa exec playwright test` | `apps/pwa/e2e/layout.spec.ts` | ⬜ pending |
|
||||||
|
| W1 | TEST-01 | Calendar populated + empty + error states (UI-SPEC Rules 4/5) | E2E | same | `apps/pwa/e2e/calendar.spec.ts` | ⬜ pending |
|
||||||
|
| W1 | TEST-01 | Lists populated + empty states (UI-SPEC Rules 4/5) | E2E | same | `apps/pwa/e2e/lists.spec.ts` | ⬜ pending |
|
||||||
|
| W1 | TEST-02 | Authenticated PWA reached via `DEV_AUTH_BYPASS` (no manual login, no OIDC mock); trace shows no SW-sourced responses | E2E | `CI=true pnpm --filter @familysync/pwa exec playwright test` | spec preconditions + trace | ⬜ pending |
|
||||||
|
|
||||||
|
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Harness Self-Validation (the harness must prove it works)
|
||||||
|
|
||||||
|
The deliverable is test infrastructure — a green suite alone does not prove it would catch a real defect. The phase is validated only when the assertions provably fail on injected defects:
|
||||||
|
|
||||||
|
1. **Tap-target failure injection** — temporarily shrink a touch target (e.g. BottomTabBar `minHeight: 20px` via `page.addStyleTag`); the ≥44px assertion MUST fail. Restore → passes. Proves `boundingBox()` measures the rendered element, not the CSS source.
|
||||||
|
2. **Overflow failure injection** — add `body { width: 2000px }` via `page.addStyleTag` before the overflow assertion; it MUST fail. Remove → passes. Proves the `scrollWidth > clientWidth` check is live.
|
||||||
|
3. **SW-block verification** — with `trace: 'on-first-retry'`, inspect one trace and confirm **zero** responses sourced from `(ServiceWorker)` (D-02 / Pitfall 15).
|
||||||
|
4. **Auth-bypass verification (manual)** — with `DEV_AUTH_BYPASS` disabled the API redirects to Authelia and specs MUST fail on missing authed content; with it enabled they pass (D-01 / SC #2).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 0 Requirements
|
||||||
|
|
||||||
|
- [ ] `apps/pwa/playwright.config.ts` — project matrix (iPhone WebKit + Pixel Chromium), `globalSetup`, optional vite-only `webServer` with `reuseExistingServer: !process.env.CI`, env-driven `baseURL`, artifact/trace config
|
||||||
|
- [ ] `apps/pwa/e2e/global-setup.ts` — `/health` readiness poll + deterministic DB seed (calendar id 10 `INSERT IGNORE` guard + list + items via `mysql2`, reset-per-run)
|
||||||
|
- [ ] `apps/pwa/vitest.config.ts` — add `exclude: ['e2e/**']` to prevent the `*.spec.ts` glob collision
|
||||||
|
- [ ] `apps/pwa/package.json` — add `@playwright/test` devDependency + `test:e2e` script
|
||||||
|
- [ ] Browser install: `pnpm --filter @familysync/pwa exec playwright install --with-deps webkit chromium`
|
||||||
|
- [ ] `apps/pwa/tsconfig`/typecheck — ensure `e2e/**` passes the `tsc --noEmit` gate
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Manual-Only Verifications
|
||||||
|
|
||||||
|
| Behavior | Requirement | Why Manual | Test Instructions |
|
||||||
|
|----------|-------------|------------|-------------------|
|
||||||
|
| Auth-bypass off → Authelia redirect | TEST-02 / SC #2 | Requires flipping `DEV_AUTH_BYPASS` off on the dev stack (env change outside the spec) | Disable bypass, run suite, confirm specs fail on missing authed content; re-enable, confirm green |
|
||||||
|
| Real prod-service-worker behavior | (out of scope) | Harness blocks the SW by design (D-02); prod SW is a device gate | Human/device check — not automated here |
|
||||||
|
| iOS-Safari standalone-PWA (Home-Screen install, standalone OIDC redirect, iOS push) | (out of scope) | Cannot be driven by Playwright/WebKit emulation | Human/device gate per project CLAUDE.md exception |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Validation Sign-Off
|
||||||
|
|
||||||
|
- [ ] All tasks have `<automated>` verify or a Wave 0 dependency
|
||||||
|
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
|
||||||
|
- [ ] Wave 0 covers all MISSING references (config, global-setup, browser install)
|
||||||
|
- [ ] No watch-mode flags in committed commands
|
||||||
|
- [ ] Feedback latency < 60s
|
||||||
|
- [ ] `nyquist_compliant: true` set in frontmatter (by planner once task map is complete)
|
||||||
|
|
||||||
|
**Approval:** pending
|
||||||
Reference in New Issue
Block a user