diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2a11f27 --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +# Secrets — every collection keeps its own .env beside its .env.example +.env +**/.env +!**/.env.example + +# MemPalace per-project files (issue #185). +# `mempalace init` ignores both by default. We deliberately track mempalace.yaml — +# it is just wing/room config, and losing it on a fresh clone would make mining +# fall back to basename auto-detection (wing "scripts", which collides). +# entities.json stays ignored: it holds detected people/entities. +entities.json diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..b516f5e --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,212 @@ +# scripts + +Operational scripts for Luc's self-hosted environment. Each script collection is an +independent, self-documenting folder at the repo root. There is no shared runtime, no +build step, and no top-level package manifest — this is a repo of standalone tools, not +an application. + +--- + +## The one hard rule: collections are self-contained + +Every script collection lives in **its own top-level folder** and carries **all of its own +documentation, configuration, and dependencies**. A collection must be understandable, +runnable, and deletable on its own. + +``` +scripts/ +├── CLAUDE.md # this file — repo-wide context +├── README.md # human-facing index of collections +├── _template/ # scaffold for a new collection (leading _ = not a collection) +└── / + ├── README.md # REQUIRED — what it does, prerequisites, usage + ├── CLAUDE.md # OPTIONAL — agent context, only if non-obvious + ├── .env.example # REQUIRED if the collection reads secrets + ├── bin/ # executable entrypoints (chmod +x, shebang) + ├── lib/ # sourced helpers, not directly executable + └── docs/ # anything longer than the README warrants +``` + +**Do not** create shared/, common/, or utils/ at the repo root. If two collections need +the same helper, copy it. Duplication across collections is explicitly preferred over +coupling — a collection that depends on a sibling is no longer independently deletable, +and these scripts get deployed to different places at different times. + +Folders prefixed with `_` are repo infrastructure, not collections. + +### Adding a collection + +Copy `_template/` to the new name, fill in its README, and add one line to the root +`README.md` index. That is the whole process. + +--- + +## The environment + +**odin** — the Unraid server; the single host everything self-hosted runs on. Reached from +this box over **Tailscale** (`100.101.253.105`, LAN `192.168.90.103`). + +> **See [docs/odin-access.md](docs/odin-access.md)** for the access path, the WebGUI/API +> endpoint, TLS caveats, and the Unraid GraphQL API — all verified from this box. + +Verified characteristics: + +| Aspect | Detail | +| --- | --- | +| Host OS | Unraid | +| Container runtime | Docker + Docker Compose (stacks defined per-app) | +| Database | MariaDB 11.x — **PostgreSQL is not available**, treat as a hard constraint | +| Cache / queue | Redis (available, used optionally) | +| Auth | Authelia — OIDC/OAuth2 provider for internal apps | +| Ingress | Pangolin/Newt tunnel — **no open inbound ports** | +| DNS | Split-DNS on `bergerhouse.net`; private IPs resolve internally | +| Git forge | Self-hosted Gitea/Forgejo at `git.bergerhouse.net` (user `luckberg`, SSH port 222) | +| CI | Gitea Actions with a self-hosted runner | +| Mail / calendar | Fastmail (paid) — JMAP/CalDAV, the source of truth for calendars | +| Remote access | Tailscale; odin `100.101.253.105`, subnet route for `192.168.90.0/24` | +| Unraid API | GraphQL at `{myunraid.net host}:8443/graphql`, `x-api-key` auth (Unraid 7.2+) | +| Automation | n8n on odin — drives the Docker-manager Telegram bot | + +Known service hostnames: `git.bergerhouse.net`, `familysync.bergerhouse.net`, +`familysync-dev.bergerhouse.net`. + +**Not on odin:** GitHub (`gh` is not installed — use `tea` for the Gitea forge), +PostgreSQL, any cloud provider. + +### This box vs. odin + +The machine this repo is edited on is `dev` (Tailscale `100.94.16.46`) — a **separate +workstation**, and in fact a VPS with a public IP, not a machine on the home LAN. Scripts +written here usually *target* odin. Never assume a script runs on the same host it was +authored on: take the target as a parameter or read it from config. + +Tailscale is the **only** path from `dev` to odin — if it is down, nothing below resolves. + +`?` — still unconfirmed: the SSH login to odin (no `~/.ssh/config` entry on this box) and +where deployed scripts are expected to live on the Unraid host. + +### Sibling repos on `git.bergerhouse.net` + +`luckberg/` holds `scripts` (this repo), `familysync`, `unraid-docker-manager`, +`azure-epac`, `_cargo-index`. Two are load-bearing context: + +- **`unraid-docker-manager`** — n8n + Telegram bot managing odin's containers. The + authoritative source on the Unraid GraphQL API and container polling; distilled into + [docs/odin-access.md](docs/odin-access.md). +- **`familysync`** — the origin of most of the stack facts in the table above. + +--- + +## Prior context: MemPalace + +Much of what is known about this environment lives in **MemPalace**, a local semantic +memory store — not in this repo. Query it before asking Luc to re-explain something, and +before assuming an infrastructure detail is undocumented. + +| Aspect | Detail | +| --- | --- | +| Version | `mempalace` v3.4.1, installed as a `uv` tool | +| Palace path | `/home/luc/.mempalace/palace` | +| Backend | ChromaDB (vector search) + SQLite (metadata); fully local, no API key | +| Access | MCP server (`mempalace-mcp`) → 19 `mempalace_*` tools; plus the `mempalace` CLI | +| Size | ~32k drawers | + +**Read the docs, don't guess the CLI.** MemPalace ships its own instructions: + +```bash +mempalace instructions +``` + +Structure is **Wings → Rooms → Closets → Drawers** — wings are projects/people, rooms are +topics, closets are summaries, drawers are verbatim memories. *Halls* connect rooms within +a wing; *tunnels* connect rooms across wings. + +Existing wings: `familysync` (~21.6k), `sessions` (~10.6k — mined agent transcripts), +`wing_familysync`. + +### This repo's wing + +Initialized with `mempalace init . --yes` and mined with `mempalace mine .`. + +- **Wing: `odin-scripts`** — set explicitly in `mempalace.yaml`, *not* auto-detected. + Auto-detection uses the directory basename, and a wing called `scripts` would collide + with any other corpus in a folder of that name. +- **Rooms:** `environment`, `documentation`, `collections`, `general`. +- Room routing matches keywords against the file **path**, so never use `scripts` as a + room keyword here — it matches this repo's own path and swallows every file. (Learned + the hard way; the comment in `mempalace.yaml` says so too.) +- `mempalace.yaml` and `entities.json` are gitignored — that is MemPalace's own + convention (issue #185), applied by `init`. + +Re-mine after adding a collection: `mempalace mine . --agent claude`. Mining is +idempotent — already-filed files are skipped. + +Practical notes: + +- `mempalace_search` is semantic — the `query` field takes **keywords only**, max 250 + chars. Put background in `context`, not in the query, or the embedding gets diluted. +- Filter with `wing`/`room` when you know where the answer lives; the `sessions` wing is + raw transcript chunks and is noisy for factual lookups. +- **BM25 scores of 0 with mid-range similarity means no lexical match** — a bare-keyword + search for a proper noun that returns only semantic neighbours is a *negative* result, + not a weak positive. This is how the `odin` gap above was established. +- `mempalace_kg_query` hits the temporal knowledge graph for point-in-time facts; prefer + it over `search` for "what is X currently" questions, and use `kg_supersede` rather + than invalidate-then-add when a single-valued fact changes. +- Auto-save hooks are active: a **Stop** hook saves every 15 human messages, and a + **PreCompact** hook force-saves before context compaction. + +Complementary and **separate** from MemPalace: + +- **Per-project agent memory** — `~/.claude/projects//memory/*.md`, indexed by that + directory's `MEMORY.md`. This repo's is currently empty; FamilySync's holds the CI-gate, + dev-stack, and Gitea-CI notes that much of this file's environment section derives from. +- **Project docs on disk** — `~/projects/familysync/CLAUDE.md` and `docs/ARCHITECTURE.md` + are the authoritative written source for the odin stack. + +--- + +## Conventions + +These apply to every collection unless its own README documents a deliberate exception. + +### Shell + +- `#!/usr/bin/env bash` and `set -euo pipefail` at the top of every bash entrypoint. +- Quote every expansion. Prefer `[[ ]]` over `[ ]`. +- Scripts must be **idempotent** — safe to re-run. Assume cron or a retry will do so. +- Accept `--dry-run` for anything that mutates state, deletes, or sends. Default to the + safe path when a flag is ambiguous. +- Log to stdout, errors to stderr. No log files unless the collection documents rotation. + +### Secrets + +- **Never commit secrets.** Read them from the environment or a file path passed in. +- Every collection that needs secrets ships a `.env.example` with dummy values and a + README line naming where the real values live. +- Note the FamilySync-side precedent: gitleaks runs as a blocking CI gate on that repo. + Assume any secret committed here is treated the same way — as a leak to be rotated, not + a mistake to be amended away. + +### Portability + +- Target the interpreters actually present: bash, Python 3, Node 22. Check before adding + a dependency on anything else. +- Unraid's userland is BusyBox-leaning in places — prefer POSIX-portable invocations of + `sed`/`awk`/`date` over GNU-only flags when the script runs on the server. + +--- + +## Working agreements for Claude + +- **Scope changes to one collection.** A request about collection X should not touch + collection Y. If it must, say so explicitly rather than doing it quietly. +- **Update the collection's README in the same change as the code.** Docs living beside + the script is the point of this repo's layout; a code change that leaves its README + stale defeats it. +- **Don't invent infrastructure.** If a script needs a host, path, port, or credential + that isn't documented above, ask — do not guess a plausible value. Wrong infra + assumptions in an ops script fail in production, not in review. +- **Destructive operations need confirmation** before they're run against odin, even when + the script itself is finished and correct. Writing the script is not authorization to + execute it. diff --git a/README.md b/README.md index e69de29..ca19142 100644 --- a/README.md +++ b/README.md @@ -0,0 +1,26 @@ +# scripts + +Operational scripts for my self-hosted environment (**odin**, an Unraid host running +Docker Compose behind Authelia and a Pangolin/Newt tunnel). + +Each script collection lives in its own top-level folder and is fully self-contained — +its own README, config, and dependencies. Collections never import from each other. + +See [CLAUDE.md](CLAUDE.md) for the environment details and repo conventions, and +[docs/odin-access.md](docs/odin-access.md) for how to reach odin and drive its API. + +## Collections + +_None yet._ + + + +## Adding a collection + +```bash +cp -r _template +``` + +Fill in its `README.md`, then add a line to the table above. diff --git a/_template/.env.example b/_template/.env.example new file mode 100644 index 0000000..f97dab7 --- /dev/null +++ b/_template/.env.example @@ -0,0 +1,3 @@ +# Copy to .env and fill in. Never commit the filled-in .env. +EXAMPLE_HOST=odin.bergerhouse.net +EXAMPLE_TOKEN=replace-me diff --git a/_template/README.md b/_template/README.md new file mode 100644 index 0000000..c35301e --- /dev/null +++ b/_template/README.md @@ -0,0 +1,36 @@ +# + +One sentence: what this collection does and why it exists. + +## Prerequisites + +- Where it runs: `odin` (Unraid) / this workstation / either +- Interpreters: bash / Python 3 / Node 22 +- External tools: e.g. `docker`, `tea`, `curl` +- Access needed: e.g. SSH to odin, Gitea API token + +## Configuration + +Copy `.env.example` to `.env` and fill it in. Real values live in ``. + +| Variable | Required | Description | +| --- | --- | --- | +| `EXAMPLE_HOST` | yes | Target host to operate against | + +## Usage + +```bash +bin/example --dry-run # preview, changes nothing +bin/example # apply +``` + +## Behaviour notes + +- Idempotent: yes/no — and what happens on a re-run. +- Destructive operations: list them, or state "none". +- Scheduling: cron entry / Unraid User Scripts / manual only. + +## Gotchas + +Anything that cost time to discover. This is the section that earns the collection its +own folder — keep it honest and current. diff --git a/_template/bin/example b/_template/bin/example new file mode 100755 index 0000000..42afc4b --- /dev/null +++ b/_template/bin/example @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# Template entrypoint. Copy, rename, and replace main(). +set -euo pipefail + +readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly COLLECTION_DIR="$(dirname "$SCRIPT_DIR")" + +DRY_RUN=0 + +usage() { + cat <<'EOF' +Usage: example [--dry-run] [--help] + + --dry-run Show what would happen without changing anything. + --help Show this message. +EOF +} + +log() { printf '%s\n' "$*"; } +warn() { printf '%s\n' "$*" >&2; } +die() { warn "error: $*"; exit 1; } + +# Run a mutating command, or describe it under --dry-run. +run() { + if (( DRY_RUN )); then + log "[dry-run] $*" + else + "$@" + fi +} + +load_env() { + local env_file="$COLLECTION_DIR/.env" + [[ -f "$env_file" ]] || return 0 + set -a; . "$env_file"; set +a +} + +main() { + while (( $# )); do + case "$1" in + --dry-run) DRY_RUN=1 ;; + --help|-h) usage; exit 0 ;; + *) die "unknown argument: $1" ;; + esac + shift + done + + load_env + : "${EXAMPLE_HOST:?EXAMPLE_HOST is not set — see .env.example}" + + log "target: $EXAMPLE_HOST" + run true # replace with the real work +} + +main "$@" diff --git a/docs/odin-access.md b/docs/odin-access.md new file mode 100644 index 0000000..108af82 --- /dev/null +++ b/docs/odin-access.md @@ -0,0 +1,149 @@ +# Accessing odin + +Reference for reaching the Unraid host (**odin**) from this workstation (`dev`) and +driving its API. Everything here was verified from `dev` on 2026-08-15; the "Verified" +column says how. + +Related repo: **`luckberg/unraid-docker-manager`** on `git.bergerhouse.net` — an n8n +Telegram bot that manages odin's containers. It is the origin of most of the API detail +below and is worth reading before writing anything new against the Unraid API. + +--- + +## Network path: Tailscale + +`dev` and odin are both on the tailnet. `dev` is a **VPS** (public IP `51.222.234.162`), +*not* a machine on the home LAN — Tailscale is the only path to odin. + +| Node | Tailscale IP | Notes | +| --- | --- | --- | +| `dev` | `100.94.16.46` | this workstation (linux) | +| `odin` | `100.101.253.105` | the Unraid host (linux) | + +Other tailnet nodes seen: `duplicati`, `experience` (windows), `fedora`, `nusgwvm`, +`salt`, `google-pixel-8-pro`. + +**A subnet route is advertised for the home LAN.** odin's LAN address `192.168.90.103` is +reachable from `dev` and routes over `tailscale0`: + +``` +$ ip route get 192.168.90.103 +192.168.90.103 dev tailscale0 table 52 src 100.94.16.46 +``` + +Both the tailnet IP and the LAN IP answer at ~34 ms RTT — the same path. This matters: +the `myunraid.net` URL below looks like a cloud relay, but **DNS resolves it to the +private `192.168.90.103`**, so traffic goes direct over Tailscale and never leaves the +tailnet. If Tailscale is down, none of it is reachable. + +--- + +## WebGUI / API endpoint + +``` +https://192-168-90-103.87c90a69c53e7197560c778a63f483e675130f2a.myunraid.net:8443/ +``` + +The hostname encodes odin's LAN IP with dashes, under a per-server hash on `myunraid.net`. +This is the **canonical vhost** — not merely a convenience alias. Use it for everything. + +### Why the plain LAN IP does not work + +Both failure modes are real and confirmed: + +| Attempt | Result | Verified | +| --- | --- | --- | +| `https://192.168.90.103:8443/graphql` | **404** from nginx | curl | +| `http://192.168.90.103/graphql` | **302** → the `myunraid.net` URL | curl | +| `https://:8443/graphql` | **200**, GraphQL responds | curl | + +nginx only serves `/graphql` on the `myunraid.net` `server_name`; the raw IP vhost returns +404. Plain HTTP redirects to that hostname — and **the redirect strips the `x-api-key` +header**, so a client that follows redirects authenticates as nobody. Always call the +`myunraid.net` URL directly rather than relying on a redirect. + +### TLS caveat — correcting the other repo + +`unraid-docker-manager` states "Valid certs via myunraid.net — no SSL ignore needed." +**That is not true from `dev`.** Verification fails here: + +``` +depth=1 C=US, O=Let's Encrypt, CN=YR1 +verify error:num=20:unable to get local issuer certificate +depth=0 CN=*.87c90a69c53e7197560c778a63f483e675130f2a.myunraid.net +``` + +The leaf is a genuine Let's Encrypt wildcard, but the chain presents an intermediate +(`YR1`) that this box's CA bundle (`ca-certificates 20250419`) cannot chain to a trusted +root — either the server omits the intermediate or the bundle predates it. The cert is not +forged; the chain is incomplete *for this client*. + +Practical consequence: **`curl` needs `-k` from `dev`**, and any script that talks to odin +must either pass the insecure flag or pin/supply the issuer. n8n running *on* odin may +well verify fine, which is likely why the other repo's claim held there. Prefer supplying +the missing intermediate over making `-k` a permanent habit — `-k` disables verification +entirely, which on a tailnet is a tolerable but real weakening. + +--- + +## Unraid GraphQL API + +- **Endpoint:** `{UNRAID_HOST}/graphql`, POST, where `UNRAID_HOST` is the `myunraid.net` + base URL **without** the `/graphql` suffix. +- **Auth:** `x-api-key: ` header. +- **Availability:** native in Unraid 7.2+ (odin is on 7.2); 6.9–7.1 needs the Connect plugin. + +Unauthenticated requests return HTTP **200** with a GraphQL error body — not an HTTP 401: + +```json +{"errors":[{"message":"Invalid CSRF token","extensions":{"code":"UNAUTHENTICATED", + "originalError":{"error":"Unauthorized","statusCode":401}}}],"data":null} +``` + +**Never treat HTTP 200 as success.** Always inspect `response.errors[]`. + +### Polling running containers + +```bash +curl -sS -k -X POST "${UNRAID_HOST}/graphql" \ + -H 'Content-Type: application/json' \ + -H "x-api-key: ${UNRAID_API_KEY}" \ + -d '{"query":"query { docker { containers { id names state } } }"}' +``` + +```json +{"data":{"docker":{"containers":[ + {"id":":","names":["/n8n"],"state":"RUNNING"} +]}}} +``` + +Field behaviours that bite: + +- `state` is **UPPERCASE** — `RUNNING`, not `running`. +- `names` is an array and entries are **`/`-prefixed** — `/n8n`, not `n8n`. +- `id` is a `PrefixedID`: `{server_hash}:{container_hash}`, two 64-char SHA-256 hex + strings joined by a colon. The server half is identical for every container on odin. +- **`isUpdateAvailable` does not exist** in the 7.2 schema, despite appearing in that + repo's earlier research. Introspect before relying on any field. + +### API keys + +Not present on this box — `.env.unraid-api` is gitignored and absent from the clone. Create +a key scoped to what the script actually needs: + +```bash +unraid-api apikey --create --name "" --permissions "DOCKER:UPDATE_ANY" --json +``` + +Or WebGUI → Settings → Management Access → API Keys. Store it in the collection's own +`.env`, never in git. + +--- + +## Gotchas carried over from `unraid-docker-manager` + +- **Each Bash tool call is a fresh shell.** Source the env file in the *same* command + chain as the request: `. .env.unraid-api; curl ...`. A separate `source` call is lost. +- **Large API responses:** save to a temp file before parsing; piping a 400 KB body + straight into `python3 -c` fails silently. +- Docker API `204 No Content` means success with an empty body. diff --git a/mempalace.yaml b/mempalace.yaml new file mode 100644 index 0000000..85da455 --- /dev/null +++ b/mempalace.yaml @@ -0,0 +1,28 @@ +# Wing is set explicitly, not auto-detected from the directory basename — +# "scripts" is a common folder name and would collide with other corpora. +wing: odin-scripts +rooms: + - name: environment + description: Repo-wide context — the odin/Unraid environment, conventions, agent guidance + keywords: + - environment + - odin + - unraid + - conventions + - CLAUDE + - name: documentation + description: Per-collection READMEs and docs/ content + keywords: + - documentation + - docs + - readme + # NB: do not use "scripts" as a keyword here — it matches the repo's own path + # (/home/luc/projects/scripts) and would swallow every file into this room. + - name: collections + description: Executable entrypoints and sourced helpers (bin/, lib/) + keywords: + - bin + - lib + - name: general + description: Files that don't fit other rooms + keywords: []