diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index a998351..5ed4e69 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -20,10 +20,13 @@ Modularize the workflow for maintainability, add "update all" functionality, fix **Requirements:** MOD-01, MOD-02, DEBT-03 -**Plans:** 0 plans +**Plans:** 4 plans Plans: -- [ ] TBD (run /gsd:plan-phase 10 to break down) +- [ ] 10-01-PLAN.md — Orphan node cleanup (remove 8 orphan nodes before modularization) +- [ ] 10-02-PLAN.md — Extract container update sub-workflow (consolidates DEBT-03) +- [ ] 10-03-PLAN.md — Extract container actions sub-workflow (start/stop/restart) +- [ ] 10-04-PLAN.md — Integration verification and user checkpoint **Success Criteria:** 1. Workflow split into logical sub-workflows (command routing, container operations, keyboard generation, etc.) @@ -104,7 +107,7 @@ Plans: | 7 | Socket Security | v1.1 | Complete | | 8 | Inline Keyboard Infrastructure | v1.1 | Complete | | 9 | Batch Operations | v1.1 | Complete | -| 10 | Workflow Modularization | v1.2 | Pending | +| 10 | Workflow Modularization | v1.2 | Planned | | 11 | Update All & Callback Limits | v1.2 | Pending | | 12 | Polish & Audit | v1.2 | Pending | | 13 | Documentation Overhaul | v1.2 | Pending | @@ -112,4 +115,4 @@ Plans: **v1.2 Coverage:** 12 requirements mapped across 4 phases --- -*Updated: 2026-02-04 after v1.1 milestone* +*Updated: 2026-02-04 after Phase 10 planning* diff --git a/.planning/phases/10-workflow-modularization/10-01-PLAN.md b/.planning/phases/10-workflow-modularization/10-01-PLAN.md new file mode 100644 index 0000000..28502a2 --- /dev/null +++ b/.planning/phases/10-workflow-modularization/10-01-PLAN.md @@ -0,0 +1,134 @@ +--- +phase: 10-workflow-modularization +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: [n8n-workflow.json] +autonomous: true + +must_haves: + truths: + - "Workflow has no orphan nodes visible in n8n canvas" + - "All existing functionality still works after cleanup" + - "Workflow node count reduced by 8" + artifacts: + - path: "n8n-workflow.json" + provides: "Cleaned workflow without orphan nodes" + contains: "Telegram Trigger" + key_links: + - from: "Telegram Trigger" + to: "All action paths" + via: "No broken connections" + pattern: "connections.*main" +--- + + +Remove 8 orphan nodes from the n8n workflow before modularization work begins. + +Purpose: Clean up vestigial nodes from workflow evolution to establish a clean baseline for modularization. Orphan nodes clutter the canvas and may cause confusion during sub-workflow extraction. + +Output: Workflow JSON with orphan nodes removed, deployed and verified working. + + + +@/home/luc/.claude/get-shit-done/workflows/execute-plan.md +@/home/luc/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/10-workflow-modularization/10-RESEARCH.md +@n8n-workflow.json + + + + + + Task 1: Identify and document orphan nodes + n8n-workflow.json + +Analyze the workflow to identify all orphan nodes - nodes with no incoming connections that are not triggers, and nodes with no outgoing connections that are not legitimate terminal nodes (like Telegram send messages). + +Programmatic analysis has found at minimum: +- "Answer Batch Exec" (position [1340, 900]) - httpRequest with no incoming connection +- "Batch Loop" (position [3100, -500]) - splitInBatches with no connections + +The user reports 8 total orphan nodes. Examine the workflow carefully to identify all 8: +1. Search for nodes with no incoming connections that aren't triggers +2. Search for nodes with no outgoing connections that aren't terminal nodes (Send/Edit messages) +3. Check positions far from main flow (negative Y positions, isolated X positions) +4. Look for vestigial nodes from prior development phases + +Document each orphan with: +- Node name +- Node type +- Position +- Why it's orphaned (no connections, leftover from development, etc.) + + Create a list of all 8 orphan nodes with their positions and types + All 8 orphan nodes identified and documented + + + + Task 2: Remove orphan nodes and deploy + n8n-workflow.json + +Remove all identified orphan nodes from n8n-workflow.json: + +1. For each orphan node: + - Remove the node object from the "nodes" array + - Remove any connection entries referencing the node from "connections" object + - Note: Use the node "name" field to find connections, not "id" + +2. Verify JSON validity after removal: + - Parse the JSON to confirm it's valid + - Check that no connections reference removed nodes + +3. Deploy updated workflow to n8n: + - Use n8n API to update the workflow + - Verify workflow activates without errors + +4. Test core functionality still works: + - Test /status command + - Test container submenu navigation + - Test at least one action (start/stop/restart) + +Do NOT remove any nodes that: +- Are triggers (Telegram Trigger) +- Are legitimate terminal nodes (Send/Edit message nodes) +- Have both incoming AND outgoing connections +- Are part of the batch execution flow (even if appears orphaned, verify first) + + +- `python3 -c "import json; json.load(open('n8n-workflow.json'))"` succeeds +- Workflow deploys via n8n API without errors +- /status command returns container list +- At least one container action works + + 8 orphan nodes removed, workflow deployed, core functionality verified working + + + + + +1. Workflow JSON is valid and parses without errors +2. n8n workflow is deployed and active +3. /status command shows container list inline keyboard +4. Container actions (start/stop/restart) work +5. Batch operations still function +6. Text commands (status, update ) still work + + + +- 8 orphan nodes removed from workflow +- Node count reduced from 248 to ~240 +- All existing bot functionality works +- Workflow ready for modularization + + + +After completion, create `.planning/phases/10-workflow-modularization/10-01-SUMMARY.md` + diff --git a/.planning/phases/10-workflow-modularization/10-02-PLAN.md b/.planning/phases/10-workflow-modularization/10-02-PLAN.md new file mode 100644 index 0000000..a6a9bd6 --- /dev/null +++ b/.planning/phases/10-workflow-modularization/10-02-PLAN.md @@ -0,0 +1,222 @@ +--- +phase: 10-workflow-modularization +plan: 02 +type: execute +wave: 2 +depends_on: [10-01] +files_modified: [n8n-workflow.json, n8n-container-update.json] +autonomous: true + +must_haves: + truths: + - "Single container update via text command works" + - "Single container update via inline keyboard works" + - "Batch update operations work" + - "Update flow exists in one place only (sub-workflow)" + artifacts: + - path: "n8n-container-update.json" + provides: "Container update sub-workflow" + contains: "executeWorkflowTrigger" + - path: "n8n-workflow.json" + provides: "Main workflow calling update sub-workflow" + contains: "executeWorkflow" + key_links: + - from: "n8n-workflow.json" + to: "n8n-container-update.json" + via: "Execute Sub-workflow node" + pattern: "executeWorkflow.*container-update" +--- + + +Extract the container update flow into a dedicated sub-workflow to consolidate duplicated code (DEBT-03). + +Purpose: The update logic is currently duplicated between the text command path (~lines 1656-2400) and the callback/inline keyboard path (~lines 3628-4010). Extracting to a sub-workflow creates a single source of truth, reduces main workflow complexity, and makes the update logic independently testable. + +Output: New container-update sub-workflow JSON file and updated main workflow that calls it from both text and callback paths. + + + +@/home/luc/.claude/get-shit-done/workflows/execute-plan.md +@/home/luc/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/phases/10-workflow-modularization/10-RESEARCH.md +@.planning/phases/10-workflow-modularization/10-01-SUMMARY.md +@n8n-workflow.json + + + + + + Task 1: Create container-update sub-workflow + n8n-container-update.json + +Create a new sub-workflow file `n8n-container-update.json` that encapsulates the entire container update flow. + +**Input contract (Execute Sub-workflow Trigger with defined fields):** +```json +{ + "containerId": "string - Docker container ID", + "containerName": "string - Container name for messages", + "chatId": "number - Telegram chat ID", + "messageId": "number - Message ID for inline edits (0 for text mode)", + "responseMode": "string - 'text' or 'inline'" +} +``` + +**Flow to extract (from research):** +1. Inspect container configuration (get current image, config, host config) +2. Pull latest image (with :latest tag protection) +3. Inspect new image and compare digests +4. If update needed: + - Stop container + - Remove old container + - Create new container with same config + - Start new container +5. Clean up old image +6. Return result + +**Output contract:** +```json +{ + "success": "boolean", + "message": "string - Result message for user", + "updated": "boolean - Whether update was performed", + "oldDigest": "string (optional)", + "newDigest": "string (optional)" +} +``` + +**Implementation notes:** +- Use "Define using fields" schema type for input contract (per research best practice) +- Include progress message editing for inline mode (messageId > 0) +- Include new message sending for text mode (messageId == 0) +- Handle both "update needed" and "already up to date" cases +- Preserve :latest tag protection (default to :latest if no tag) +- Preserve image cleanup after successful update + + +- JSON file is valid: `python3 -c "import json; json.load(open('n8n-container-update.json'))"` +- Contains "executeWorkflowTrigger" node +- Has proper input schema with all 5 fields + + Container update sub-workflow created with proper input/output contracts + + + + Task 2: Wire main workflow to use sub-workflow + n8n-workflow.json + +Modify the main workflow to call the container-update sub-workflow instead of having inline update logic. + +**Changes needed:** + +1. **Text command path:** + - Find the text update flow (around the "Update Container" text command routing) + - Replace inline update nodes with: + a. Code node to prepare sub-workflow input (containerId, containerName, chatId, messageId=0, responseMode='text') + b. Execute Sub-workflow node pointing to container-update workflow + - Remove the duplicate update logic nodes from text path + +2. **Callback/inline path:** + - Find the callback update flow (around "Handle Update Action" or similar) + - Replace inline update nodes with: + a. Code node to prepare sub-workflow input (containerId, containerName, chatId, messageId from callback, responseMode='inline') + b. Execute Sub-workflow node pointing to container-update workflow + - Remove the duplicate update logic nodes from callback path + +3. **Batch update path:** + - Find where batch update calls individual container updates + - Ensure it also uses the sub-workflow (may already be structured to call the same code) + +**Execute Sub-workflow node configuration:** +```json +{ + "parameters": { + "source": "database", + "workflowId": "", + "mode": "once", + "options": { + "waitForSubWorkflow": true + } + }, + "type": "n8n-nodes-base.executeWorkflow" +} +``` + +**Important:** After extraction, the main workflow should be significantly shorter (estimate ~750 fewer lines per research). + + +- Main workflow JSON is valid +- Contains "executeWorkflow" node(s) for update paths +- Old inline update nodes are removed (workflow is shorter) + + Main workflow updated to call container-update sub-workflow from all update paths + + + + Task 3: Deploy and verify update functionality + n8n-workflow.json, n8n-container-update.json + +Deploy both workflows to n8n and verify all update paths work. + +**Deployment steps:** +1. Import container-update sub-workflow to n8n via API (create new workflow) +2. Note the workflow ID assigned by n8n +3. Update main workflow's Execute Sub-workflow node(s) with correct workflow ID +4. Deploy main workflow update to n8n via API + +**Verification tests:** + +1. **Text command update:** + - Send "update " to bot + - Should show update confirmation prompt + - Confirm and verify update completes (or shows "already up to date") + +2. **Inline keyboard update:** + - Use /status to get container list + - Select a container + - Tap "Update" button + - Should show confirmation dialog + - Confirm and verify update completes with progress messages + +3. **Batch update (if time permits):** + - Initiate a batch update for 2 containers + - Verify both update correctly + + +- Sub-workflow imported and has valid ID in n8n +- Main workflow deployed with correct sub-workflow reference +- Text "update " command works +- Inline keyboard update flow works +- Progress messages display correctly + + Update sub-workflow deployed and all update paths verified working + + + + + +1. n8n-container-update.json exists and is valid JSON +2. Sub-workflow has proper Execute Sub-workflow Trigger with input schema +3. Main workflow contains Execute Sub-workflow nodes calling update workflow +4. Main workflow line count reduced by ~500+ lines +5. Text command "update " works end-to-end +6. Inline keyboard update with confirmation works end-to-end +7. "Already up to date" case handled correctly +8. Old image cleanup still occurs after successful update + + + +- Container update logic exists in ONE place (sub-workflow) +- Both text and inline update paths use the sub-workflow +- DEBT-03 (duplicated update flow) resolved +- All update functionality works as before + + + +After completion, create `.planning/phases/10-workflow-modularization/10-02-SUMMARY.md` + diff --git a/.planning/phases/10-workflow-modularization/10-03-PLAN.md b/.planning/phases/10-workflow-modularization/10-03-PLAN.md new file mode 100644 index 0000000..3bb0d99 --- /dev/null +++ b/.planning/phases/10-workflow-modularization/10-03-PLAN.md @@ -0,0 +1,224 @@ +--- +phase: 10-workflow-modularization +plan: 03 +type: execute +wave: 2 +depends_on: [10-01] +files_modified: [n8n-workflow.json, n8n-container-actions.json] +autonomous: true + +must_haves: + truths: + - "Container start via text and inline works" + - "Container stop via text and inline works" + - "Container restart via text and inline works" + - "Simple actions exist in one place (sub-workflow)" + artifacts: + - path: "n8n-container-actions.json" + provides: "Container actions sub-workflow (start/stop/restart)" + contains: "executeWorkflowTrigger" + - path: "n8n-workflow.json" + provides: "Main workflow calling actions sub-workflow" + contains: "executeWorkflow" + key_links: + - from: "n8n-workflow.json" + to: "n8n-container-actions.json" + via: "Execute Sub-workflow node" + pattern: "executeWorkflow.*container-actions" +--- + + +Extract container simple actions (start/stop/restart) into a dedicated sub-workflow. + +Purpose: Like the update flow, start/stop/restart actions are handled in both text command and callback paths. Extracting to a sub-workflow creates a single source of truth for container state changes and supports MOD-01/MOD-02 requirements. + +Output: New container-actions sub-workflow JSON file and updated main workflow that calls it from both text and callback paths. + + + +@/home/luc/.claude/get-shit-done/workflows/execute-plan.md +@/home/luc/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/phases/10-workflow-modularization/10-RESEARCH.md +@.planning/phases/10-workflow-modularization/10-01-SUMMARY.md +@n8n-workflow.json + + + + + + Task 1: Create container-actions sub-workflow + n8n-container-actions.json + +Create a new sub-workflow file `n8n-container-actions.json` that encapsulates container start/stop/restart operations. + +**Input contract (Execute Sub-workflow Trigger with defined fields):** +```json +{ + "containerId": "string - Docker container ID", + "containerName": "string - Container name for messages", + "action": "string - 'start' | 'stop' | 'restart'", + "chatId": "number - Telegram chat ID", + "messageId": "number - Message ID for inline edits (0 for text mode)", + "responseMode": "string - 'text' or 'inline'" +} +``` + +**Flow to implement:** +1. Route based on action type (Switch node) +2. For each action: + - Call Docker API endpoint (/containers/{id}/start, /stop, /restart) + - Handle success response + - Handle error response +3. Format result message +4. Return result (or send message directly if simpler) + +**Output contract:** +```json +{ + "success": "boolean", + "message": "string - Result message for user", + "action": "string - Which action was performed", + "containerName": "string" +} +``` + +**Implementation notes:** +- Use "Define using fields" schema type for input contract +- Stop and restart require confirmation in the caller (main workflow handles confirmation dialogs) +- This sub-workflow executes the action AFTER confirmation is received +- Include proper error handling for API failures +- Message formatting should match existing bot style (emoji + container name + result) + + +- JSON file is valid: `python3 -c "import json; json.load(open('n8n-container-actions.json'))"` +- Contains "executeWorkflowTrigger" node +- Has proper input schema with all 6 fields +- Has Switch node routing to start/stop/restart paths + + Container actions sub-workflow created with proper input/output contracts + + + + Task 2: Wire main workflow to use actions sub-workflow + n8n-workflow.json + +Modify the main workflow to call the container-actions sub-workflow for start/stop/restart operations. + +**Changes needed:** + +1. **Text command path (immediate actions - start/restart):** + - Find where text commands route to start/restart handlers + - Replace inline action execution with: + a. Code node to prepare sub-workflow input + b. Execute Sub-workflow node pointing to container-actions workflow + - Keep confirmation handling in main workflow (stop still needs confirmation) + +2. **Text command path (stop with confirmation):** + - Keep confirmation dialog handling in main workflow + - After confirmation received, call sub-workflow with action='stop' + +3. **Callback/inline path (immediate actions):** + - Find where callback routes to start/restart handlers + - Replace inline execution with Execute Sub-workflow call + +4. **Callback/inline path (stop with confirmation):** + - Keep confirmation keyboard generation in main workflow + - After confirmation callback received, call sub-workflow with action='stop' + +5. **Batch action path:** + - Find where batch operations execute individual actions + - Route through the sub-workflow for consistency + +**Key principle:** Confirmation dialogs stay in main workflow. Only the actual Docker API call moves to sub-workflow. + +**Execute Sub-workflow node configuration:** +```json +{ + "parameters": { + "source": "database", + "workflowId": "", + "mode": "once", + "options": { + "waitForSubWorkflow": true + } + }, + "type": "n8n-nodes-base.executeWorkflow" +} +``` + + +- Main workflow JSON is valid +- Contains "executeWorkflow" node(s) for action paths +- Confirmation dialogs still work (handled in main workflow) + + Main workflow updated to call container-actions sub-workflow + + + + Task 3: Deploy and verify action functionality + n8n-workflow.json, n8n-container-actions.json + +Deploy the actions sub-workflow and updated main workflow, then verify all action paths work. + +**Deployment steps:** +1. Import container-actions sub-workflow to n8n via API +2. Note the workflow ID assigned by n8n +3. Update main workflow's Execute Sub-workflow node(s) with correct workflow ID +4. Deploy main workflow update to n8n via API + +**Verification tests:** + +1. **Text command actions:** + - "start " - Should start and confirm + - "stop " - Should prompt for confirmation, then stop + - "restart " - Should restart and confirm + +2. **Inline keyboard actions:** + - Use /status and select a container + - Test Start button (on stopped container) + - Test Stop button (should show confirmation) + - Test Restart button + +3. **Batch actions:** + - Select 2 containers for batch stop + - Verify confirmation and execution work + + +- Sub-workflow imported and has valid ID in n8n +- Main workflow deployed with correct sub-workflow reference +- Text commands work: start, stop (with confirmation), restart +- Inline buttons work: Start, Stop (with confirmation), Restart +- Batch operations work for start/stop/restart + + Actions sub-workflow deployed and all action paths verified working + + + + + +1. n8n-container-actions.json exists and is valid JSON +2. Sub-workflow has proper Execute Sub-workflow Trigger with input schema +3. Main workflow contains Execute Sub-workflow nodes for actions +4. Text command "start " works +5. Text command "stop " with confirmation works +6. Text command "restart " works +7. Inline keyboard Start/Stop/Restart buttons work +8. Stop confirmation dialog works in both text and inline modes +9. Batch start/stop/restart operations work + + + +- Container actions (start/stop/restart) exist in ONE place (sub-workflow) +- Both text and inline action paths use the sub-workflow +- Confirmation dialogs still function correctly +- All action functionality works as before + + + +After completion, create `.planning/phases/10-workflow-modularization/10-03-SUMMARY.md` + diff --git a/.planning/phases/10-workflow-modularization/10-04-PLAN.md b/.planning/phases/10-workflow-modularization/10-04-PLAN.md new file mode 100644 index 0000000..f0de718 --- /dev/null +++ b/.planning/phases/10-workflow-modularization/10-04-PLAN.md @@ -0,0 +1,216 @@ +--- +phase: 10-workflow-modularization +plan: 04 +type: execute +wave: 3 +depends_on: [10-02, 10-03] +files_modified: [n8n-workflow.json] +autonomous: false + +must_haves: + truths: + - "All text commands work after modularization" + - "All inline keyboard flows work after modularization" + - "All batch operations work after modularization" + - "Main workflow is significantly smaller than before" + artifacts: + - path: "n8n-workflow.json" + provides: "Modularized main workflow" + min_lines: 4000 + - path: "n8n-container-update.json" + provides: "Update sub-workflow" + contains: "executeWorkflowTrigger" + - path: "n8n-container-actions.json" + provides: "Actions sub-workflow" + contains: "executeWorkflowTrigger" + key_links: + - from: "n8n-workflow.json" + to: "n8n-container-update.json" + via: "Execute Sub-workflow" + pattern: "executeWorkflow" + - from: "n8n-workflow.json" + to: "n8n-container-actions.json" + via: "Execute Sub-workflow" + pattern: "executeWorkflow" +--- + + +Perform full integration verification of modularized workflow and checkpoint with user. + +Purpose: After extracting update and actions to sub-workflows, verify the entire bot still works correctly. This includes edge cases and flows that may not have been explicitly tested in prior plans. + +Output: Verified working modularized workflow, user confirmation, and updated ROADMAP showing phase complete. + + + +@/home/luc/.claude/get-shit-done/workflows/execute-plan.md +@/home/luc/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/phases/10-workflow-modularization/10-RESEARCH.md +@.planning/phases/10-workflow-modularization/10-01-SUMMARY.md +@.planning/phases/10-workflow-modularization/10-02-SUMMARY.md +@.planning/phases/10-workflow-modularization/10-03-SUMMARY.md +@n8n-workflow.json +@n8n-container-update.json +@n8n-container-actions.json + + + + + + Task 1: Comprehensive functionality audit + n8n-workflow.json, n8n-container-update.json, n8n-container-actions.json + +Run through all bot functionality systematically to verify nothing was broken by modularization. + +**Text command tests:** +1. `/start` or `/status` - Should show help or status +2. `status` - Should list all containers with inline keyboard +3. `status ` - Should show specific container status +4. `start ` - Should start and confirm +5. `stop ` - Should prompt confirmation, then stop +6. `restart ` - Should restart and confirm +7. `update ` - Should prompt confirmation, show progress, complete +8. `logs ` - Should show logs +9. `logs 100` - Should show 100 lines of logs +10. `batch start ` - Should batch start +11. `batch stop ` - Should batch stop with confirmation +12. `batch update ` - Should batch update with confirmation + +**Inline keyboard tests:** +1. Container list navigation (pagination if >10 containers) +2. Container submenu display +3. Start button (on stopped container) +4. Stop button with confirmation dialog +5. Restart button +6. Update button with confirmation dialog +7. Update progress messages +8. Logs button with refresh +9. Back navigation +10. Batch selection mode +11. Batch execution with progress + +**Edge cases:** +1. Container not found (fuzzy match suggestions) +2. Ambiguous container name (disambiguation) +3. Update when already up to date +4. Action on already running/stopped container +5. Confirmation timeout (30 seconds) + +Document any issues found. + + +- All text commands execute without errors +- All inline keyboard flows work +- Batch operations complete successfully +- Edge cases handled gracefully + + Full functionality audit completed, issues documented + + + + Task 2: Measure modularization impact + n8n-workflow.json, n8n-container-update.json, n8n-container-actions.json + +Quantify the improvements from modularization: + +1. **Line count comparison:** + - Original main workflow: ~8,485 lines + - New main workflow: Count lines + - Update sub-workflow: Count lines + - Actions sub-workflow: Count lines + - Calculate total and reduction percentage + +2. **Node count comparison:** + - Original: ~248 nodes (after orphan cleanup: ~240) + - New main workflow: Count nodes + - Update sub-workflow: Count nodes + - Actions sub-workflow: Count nodes + +3. **Code duplication analysis:** + - Before: Update flow duplicated (text + callback paths) + - After: Single update flow in sub-workflow + - Document specific duplication eliminated + +4. **Document the modular structure:** + ``` + Main Workflow (n8n-workflow.json) + ├── Telegram Trigger + ├── Authentication + ├── Command Routing + ├── Confirmation Dialogs + └── Sub-workflow Calls + ├── container-update (for all update operations) + └── container-actions (for start/stop/restart) + ``` + +Create a summary table of before/after metrics. + + +- Line counts documented for all workflow files +- Node counts documented +- Reduction percentage calculated +- Modular structure documented + + Modularization impact measured and documented + + + + +Modularized n8n workflow with: +- Main workflow calling sub-workflows for container operations +- Container Update sub-workflow (handles all update paths) +- Container Actions sub-workflow (handles start/stop/restart) +- Orphan nodes cleaned up +- Duplicated update code consolidated (DEBT-03) + + +Please test the following in Telegram: + +1. **Basic commands:** + - Send `status` - Should show container list keyboard + - Send `update ` - Should prompt confirmation, then update + +2. **Inline keyboard flow:** + - Tap a container from the list + - Try Start/Stop/Restart buttons + - Try Update button (with confirmation) + - Try Logs button + +3. **Batch operation:** + - Start batch mode and select 2 containers + - Execute a batch action + +Report any issues or confirm all functionality works as expected. + + Type "approved" to complete Phase 10, or describe any issues found + + + + + +1. All text commands work correctly +2. All inline keyboard flows work correctly +3. All batch operations work correctly +4. Edge cases handled (not found, disambiguation, timeouts) +5. Main workflow line count reduced significantly +6. Update flow exists in single location (sub-workflow) +7. Actions flow exists in single location (sub-workflow) +8. User has verified bot works from their phone + + + +- MOD-01: Main workflow broken into logical sub-workflows (update, actions) +- MOD-02: Sub-workflows callable from main without duplication +- DEBT-03: Update flow consolidated (no longer duplicated) +- All v1.1 functionality preserved +- User verification passed + + + +After completion, create `.planning/phases/10-workflow-modularization/10-04-SUMMARY.md` + diff --git a/.planning/phases/10-workflow-modularization/10-RESEARCH.md b/.planning/phases/10-workflow-modularization/10-RESEARCH.md new file mode 100644 index 0000000..ed7816c --- /dev/null +++ b/.planning/phases/10-workflow-modularization/10-RESEARCH.md @@ -0,0 +1,354 @@ +# Phase 10: Workflow Modularization - Research + +**Researched:** 2026-02-04 +**Domain:** n8n workflow modularization +**Confidence:** HIGH (verified against official n8n documentation) + +## Summary + +The current n8n workflow (`n8n-workflow.json`) is 8,485 lines with approximately 150+ nodes handling all functionality in a single monolithic workflow. This phase will break it into modular sub-workflows using n8n's built-in sub-workflow infrastructure. + +n8n provides first-class support for workflow modularization through the **Execute Sub-workflow** node and **Execute Sub-workflow Trigger** node. Sub-workflows enable: +- Breaking large workflows into focused components (recommended: 5-10 nodes per sub-workflow) +- Reusing logic across multiple entry points (e.g., update flow used by text commands AND callbacks) +- Memory isolation - sub-workflow memory is released after completion +- Independent testing and debugging +- Clear data contracts between components + +**Primary recommendation:** Use n8n's Execute Sub-workflow node to extract logical modules, starting with the duplicated update flow (DEBT-03) as the first extraction target since it's called from two places. + +## Standard Stack + +### Core n8n Nodes for Modularization + +| Node | Purpose | Why Standard | +|------|---------|--------------| +| Execute Sub-workflow Trigger | Entry point for sub-workflows | Built-in n8n node, defines input contract | +| Execute Sub-workflow | Calls sub-workflows from parent | Built-in n8n node, handles data passing | +| Code Node | Data transformation between modules | Prepare inputs, format outputs | +| Switch Node | Route to different sub-workflows | Handle multiple action types | + +### Input Data Modes + +| Mode | When to Use | +|------|-------------| +| Define using fields | Best for typed, validated inputs (recommended) | +| Define using JSON example | When input structure is complex/nested | +| Accept all data | Legacy/migration - avoid for new sub-workflows | + +**Recommendation:** Use "Define using fields" for all new sub-workflows - provides clear data contracts and automatic field population in the Execute Sub-workflow node. + +## Architecture Patterns + +### Recommended Module Structure + +Based on workflow analysis, these logical sub-workflows are recommended: + +``` +Main Orchestrator (n8n-workflow.json - reduced) +├── Authentication check (inline - too small to extract) +├── Command routing (inline - just a switch node) +│ +├── Sub-workflows: +│ ├── container-operations/ +│ │ ├── container-action.json # start/stop/restart a container +│ │ ├── container-update.json # full update flow (DEBT-03 consolidation) +│ │ └── container-logs.json # fetch and format logs +│ │ +│ ├── keyboard-generation/ +│ │ ├── container-list.json # paginated container list keyboard +│ │ ├── container-submenu.json # single container action menu +│ │ └── confirmation-dialog.json # stop/update confirmation keyboards +│ │ +│ └── batch-operations/ +│ ├── batch-loop.json # execute action on multiple containers +│ └── batch-selection.json # selection keyboard management +``` + +### Pattern 1: Container Update Sub-workflow (Priority - addresses DEBT-03) + +**What:** Extract the entire container update sequence into a single sub-workflow +**Why:** This logic is currently duplicated between: +1. Text command path (lines ~1656-2400) +2. Callback/inline keyboard path (lines ~3628-4010) + +**Input contract (Execute Sub-workflow Trigger):** +```json +{ + "containerId": "string", + "containerName": "string", + "chatId": "number", + "messageId": "number (optional - for inline updates)", + "responseMode": "string ('text' | 'inline')" +} +``` + +**Output contract (returned to parent):** +```json +{ + "success": "boolean", + "message": "string", + "oldDigest": "string (optional)", + "newDigest": "string (optional)" +} +``` + +**Flow extracted:** +1. Inspect container configuration +2. Pull latest image +3. Compare digests +4. If update needed: stop -> remove -> create -> start +5. Clean up old image +6. Return result + +### Pattern 2: Keyboard Generation Sub-workflow + +**What:** Extract keyboard building logic +**Why:** Keyboard generation is scattered throughout and could be centralized + +**Example input:** +```json +{ + "type": "submenu", + "containerName": "string", + "containerState": "running|exited|paused", + "options": { + "showBack": true, + "backTarget": "list:0" + } +} +``` + +### Pattern 3: Data Contract Design + +**Principle:** Every sub-workflow should have clearly defined inputs and outputs. + +``` +[Parent Workflow] + │ + ▼ Input: { containerId, containerName, chatId, messageId } +[Execute Sub-workflow Node] + │ + ▼ Processes via Execute Sub-workflow Trigger +[Sub-workflow: container-update] + │ + ▼ Output: { success, message, details } +[Parent continues with result] +``` + +### Anti-Patterns to Avoid + +- **Shared state via workflow static data:** Use explicit input/output instead +- **Over-modularization:** Don't extract nodes that are only used once +- **Deep nesting:** Keep sub-workflow call depth to 2 maximum (parent -> child) +- **Accept all data mode:** Always define explicit inputs for maintainability + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Workflow-to-workflow communication | Custom webhooks | Execute Sub-workflow node | Built-in, handles data passing, memory isolation | +| Reusable logic extraction | Copy-paste nodes | Sub-workflow with trigger | Single source of truth, DRY | +| Complex routing | Deeply nested If nodes | Switch + sub-workflows | Cleaner, testable | +| Input validation | Manual checks in Code node | Define input fields in trigger | Type safety, documentation | + +**Key insight:** n8n's sub-workflow infrastructure handles all the complexity of data passing, execution context, and memory management. Using Execute Sub-workflow node instead of custom solutions provides memory isolation that prevents heap exhaustion on large operations. + +## Common Pitfalls + +### Pitfall 1: Circular Sub-workflow Calls +**What goes wrong:** Sub-workflow A calls B which calls A, causing infinite loop +**Why it happens:** Accidental dependency cycles during refactoring +**How to avoid:** Document call hierarchy, keep sub-workflows single-purpose +**Warning signs:** Workflow hangs indefinitely, memory usage spikes + +### Pitfall 2: Lost Context in Multi-step Flows +**What goes wrong:** Data from early nodes not available in sub-workflow +**Why it happens:** Sub-workflow only receives explicit inputs, not parent context +**How to avoid:** Design complete input contracts - if sub-workflow needs data, it must be in input +**Warning signs:** "Cannot read property of undefined" errors in sub-workflow + +### Pitfall 3: Credential Scope Issues +**What goes wrong:** Sub-workflow can't access credentials defined in parent +**Why it happens:** Credentials are workflow-scoped, not globally shared +**How to avoid:** Configure same credentials in both parent and sub-workflows +**Warning signs:** Authentication failures only when running as sub-workflow + +### Pitfall 4: Breaking Existing Callback Data +**What goes wrong:** Inline keyboard callbacks stop working after modularization +**Why it happens:** Callback data parsing expects specific node structure/names +**How to avoid:** Preserve callback data format, update routing to call sub-workflows +**Warning signs:** "Callback query expired" or routing errors after deployment + +### Pitfall 5: Over-extraction +**What goes wrong:** Simple operations become 5+ sub-workflow calls, adding latency +**Why it happens:** Applying microservice patterns too aggressively +**How to avoid:** Only extract logic that is: (a) reused, OR (b) large/complex, OR (c) needs isolation +**Warning signs:** Simple operations now take seconds instead of milliseconds + +## Code Examples + +### Example 1: Sub-workflow Trigger Setup + +```json +{ + "name": "Container Update", + "nodes": [ + { + "parameters": { + "inputSource": "passthrough", + "schema": { + "schemaType": "fromFields", + "fields": [ + { "fieldName": "containerId", "fieldType": "string" }, + { "fieldName": "containerName", "fieldType": "string" }, + { "fieldName": "chatId", "fieldType": "number" }, + { "fieldName": "messageId", "fieldType": "number" }, + { "fieldName": "responseMode", "fieldType": "string" } + ] + } + }, + "name": "When executed by another workflow", + "type": "n8n-nodes-base.executeWorkflowTrigger", + "typeVersion": 1.1 + } + ] +} +``` + +### Example 2: Calling Sub-workflow from Parent + +```json +{ + "parameters": { + "source": "database", + "workflowId": "container-update-workflow-id", + "mode": "once", + "options": { + "waitForSubWorkflow": true + } + }, + "name": "Execute Update", + "type": "n8n-nodes-base.executeWorkflow", + "typeVersion": 1.2 +} +``` + +### Example 3: Preparing Sub-workflow Input (Code Node) + +```javascript +// In parent workflow, prepare input for sub-workflow +const containerData = $('Match Container').item.json; +const triggerData = $('Telegram Trigger').item.json; + +return { + json: { + containerId: containerData.matches[0].Id, + containerName: containerData.matches[0].Name, + chatId: triggerData.message.chat.id, + messageId: triggerData.message.message_id, + responseMode: 'text' + } +}; +``` + +### Example 4: Converting Callback Path to Use Sub-workflow + +```javascript +// Instead of duplicated update logic, call sub-workflow +const callbackData = $('Parse Callback Data').item.json; + +return { + json: { + containerId: callbackData.containerId, + containerName: callbackData.containerName, + chatId: callbackData.chatId, + messageId: callbackData.messageId, + responseMode: 'inline' // Sub-workflow handles response format + } +}; +``` + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| Monolithic workflows | Modular sub-workflows | n8n 1.0+ | Better maintainability, memory efficiency | +| Copy-paste nodes | Execute Sub-workflow node | Core feature | DRY, single source of truth | +| Accept all data | Define using fields | 2025 best practice | Type safety, documentation | +| Manual webhook calls | Native sub-workflow execution | Core feature | Built-in context, error handling | + +**Current n8n sub-workflow features (2026):** +- Execute Sub-workflow Trigger node with schema definition +- Sub-workflow conversion from canvas context menu (select nodes -> right-click -> Create sub-workflow) +- Memory isolation for sub-workflows +- Sub-workflow executions don't count toward n8n Cloud quotas +- Wait for completion toggle for async patterns + +## Open Questions + +### 1. Sub-workflow File Organization +**What we know:** n8n stores workflows as individual entities in its database +**What's unclear:** Whether to export sub-workflows as separate JSON files in git or keep them n8n-internal +**Recommendation:** Export as separate files for version control, document workflow IDs in main workflow + +### 2. Testing Sub-workflows +**What we know:** Can use Manual Trigger + test data for isolated testing +**What's unclear:** Best practice for automated testing +**Recommendation:** Create test input nodes in each sub-workflow, manually validate before deployment + +### 3. Migration Strategy +**What we know:** Can convert existing nodes to sub-workflow via context menu +**What's unclear:** Whether to do incremental migration or big-bang refactor +**Recommendation:** Incremental - extract one sub-workflow at a time, test, then proceed + +## Specific Extraction Plan + +Based on workflow analysis, recommended extraction order: + +### Priority 1: Container Update (addresses DEBT-03) +- **Why first:** Addresses explicit tech debt requirement, duplicated in two places +- **Lines affected:** ~1656-2400 (text path) + ~3628-4010 (callback path) +- **Expected reduction:** ~750 lines from main workflow +- **Risk:** Medium - update flow is critical, needs thorough testing + +### Priority 2: Container Simple Actions (start/stop/restart) +- **Why second:** Simpler than update, similar pattern of text+callback duplication +- **Expected reduction:** ~200 lines +- **Risk:** Low - simple operations + +### Priority 3: Keyboard Generation +- **Why third:** Scattered throughout, centralizing improves consistency +- **Expected reduction:** ~300 lines +- **Risk:** Low - UI only, easy to test + +### Priority 4: Batch Loop (if time permits) +- **Why last:** Complex state management, higher risk +- **Expected reduction:** ~400 lines +- **Risk:** Medium-High - batch operations have edge cases + +## Sources + +### Primary (HIGH confidence) +- [n8n Sub-workflows Documentation](https://docs.n8n.io/flow-logic/subworkflows/) - Official docs on sub-workflow concepts +- [Execute Sub-workflow Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/) - Official node documentation +- [Execute Sub-workflow Trigger](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflowtrigger/) - Official trigger documentation + +### Secondary (MEDIUM confidence) +- [n8n Best Practices 2026](https://michaelitoback.com/n8n-workflow-best-practices/) - Community best practices guide +- [When to Use Sub-workflows](https://community.n8n.io/t/how-to-use-sub-workflows-in-n8-when-to-use-examples/257577) - n8n Community guidance +- [n8n Expert Best Practices](https://n8n.expert/it-automation/best-practices-designing-n8n-workflows/) - Expert recommendations + +### Tertiary (LOW confidence) +- [Modularizing n8n Workflows](https://optimizesmart.com/blog/modularizing-n8n-workflows-build-smarter-workflows/) - Blog post on modularization patterns + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH - Based on official n8n documentation +- Architecture patterns: MEDIUM - Based on workflow analysis + best practices +- Pitfalls: MEDIUM - Community patterns, verified against documentation +- Extraction plan: MEDIUM - Based on codebase analysis + +**Research date:** 2026-02-04 +**Valid until:** 2026-03-04 (30 days - n8n stable, patterns unlikely to change)