docs(10): create phase plan for workflow modularization

Phase 10: Workflow Modularization - 4 plan(s) in 3 wave(s) - Wave 1: Orphan cleanup (1 plan) - Wave 2: Sub-workflow extraction (2 plans parallel) - Wave 3: Integration verification (1 plan) - Ready for execution Plans: - 10-01: Remove 8 orphan nodes - 10-02: Extract container-update sub-workflow (DEBT-03) - 10-03: Extract container-actions sub-workflow - 10-04: Integration verification with user checkpoint Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-04 11:39:54 -05:00
parent a8e947ec95
commit c122803fad
6 changed files with 1157 additions and 4 deletions
@@ -0,0 +1,354 @@
+# Phase 10: Workflow Modularization - Research
+
+**Researched:** 2026-02-04
+**Domain:** n8n workflow modularization
+**Confidence:** HIGH (verified against official n8n documentation)
+
+## Summary
+
+The current n8n workflow (`n8n-workflow.json`) is 8,485 lines with approximately 150+ nodes handling all functionality in a single monolithic workflow. This phase will break it into modular sub-workflows using n8n's built-in sub-workflow infrastructure.
+
+n8n provides first-class support for workflow modularization through the **Execute Sub-workflow** node and **Execute Sub-workflow Trigger** node. Sub-workflows enable:
+- Breaking large workflows into focused components (recommended: 5-10 nodes per sub-workflow)
+- Reusing logic across multiple entry points (e.g., update flow used by text commands AND callbacks)
+- Memory isolation - sub-workflow memory is released after completion
+- Independent testing and debugging
+- Clear data contracts between components
+
+**Primary recommendation:** Use n8n's Execute Sub-workflow node to extract logical modules, starting with the duplicated update flow (DEBT-03) as the first extraction target since it's called from two places.
+
+## Standard Stack
+
+### Core n8n Nodes for Modularization
+
+| Node | Purpose | Why Standard |
+|------|---------|--------------|
+| Execute Sub-workflow Trigger | Entry point for sub-workflows | Built-in n8n node, defines input contract |
+| Execute Sub-workflow | Calls sub-workflows from parent | Built-in n8n node, handles data passing |
+| Code Node | Data transformation between modules | Prepare inputs, format outputs |
+| Switch Node | Route to different sub-workflows | Handle multiple action types |
+
+### Input Data Modes
+
+| Mode | When to Use |
+|------|-------------|
+| Define using fields | Best for typed, validated inputs (recommended) |
+| Define using JSON example | When input structure is complex/nested |
+| Accept all data | Legacy/migration - avoid for new sub-workflows |
+
+**Recommendation:** Use "Define using fields" for all new sub-workflows - provides clear data contracts and automatic field population in the Execute Sub-workflow node.
+
+## Architecture Patterns
+
+### Recommended Module Structure
+
+Based on workflow analysis, these logical sub-workflows are recommended:
+
+```
+Main Orchestrator (n8n-workflow.json - reduced)
+├── Authentication check (inline - too small to extract)
+├── Command routing (inline - just a switch node)
+│
+├── Sub-workflows:
+│   ├── container-operations/
+│   │   ├── container-action.json      # start/stop/restart a container
+│   │   ├── container-update.json      # full update flow (DEBT-03 consolidation)
+│   │   └── container-logs.json        # fetch and format logs
+│   │
+│   ├── keyboard-generation/
+│   │   ├── container-list.json        # paginated container list keyboard
+│   │   ├── container-submenu.json     # single container action menu
+│   │   └── confirmation-dialog.json   # stop/update confirmation keyboards
+│   │
+│   └── batch-operations/
+│       ├── batch-loop.json            # execute action on multiple containers
+│       └── batch-selection.json       # selection keyboard management
+```
+
+### Pattern 1: Container Update Sub-workflow (Priority - addresses DEBT-03)
+
+**What:** Extract the entire container update sequence into a single sub-workflow
+**Why:** This logic is currently duplicated between:
+1. Text command path (lines ~1656-2400)
+2. Callback/inline keyboard path (lines ~3628-4010)
+
+**Input contract (Execute Sub-workflow Trigger):**
+```json
+{
+  "containerId": "string",
+  "containerName": "string",
+  "chatId": "number",
+  "messageId": "number (optional - for inline updates)",
+  "responseMode": "string ('text' | 'inline')"
+}
+```
+
+**Output contract (returned to parent):**
+```json
+{
+  "success": "boolean",
+  "message": "string",
+  "oldDigest": "string (optional)",
+  "newDigest": "string (optional)"
+}
+```
+
+**Flow extracted:**
+1. Inspect container configuration
+2. Pull latest image
+3. Compare digests
+4. If update needed: stop -> remove -> create -> start
+5. Clean up old image
+6. Return result
+
+### Pattern 2: Keyboard Generation Sub-workflow
+
+**What:** Extract keyboard building logic
+**Why:** Keyboard generation is scattered throughout and could be centralized
+
+**Example input:**
+```json
+{
+  "type": "submenu",
+  "containerName": "string",
+  "containerState": "running|exited|paused",
+  "options": {
+    "showBack": true,
+    "backTarget": "list:0"
+  }
+}
+```
+
+### Pattern 3: Data Contract Design
+
+**Principle:** Every sub-workflow should have clearly defined inputs and outputs.
+
+```
+[Parent Workflow]
+    │
+    ▼ Input: { containerId, containerName, chatId, messageId }
+[Execute Sub-workflow Node]
+    │
+    ▼ Processes via Execute Sub-workflow Trigger
+[Sub-workflow: container-update]
+    │
+    ▼ Output: { success, message, details }
+[Parent continues with result]
+```
+
+### Anti-Patterns to Avoid
+
+- **Shared state via workflow static data:** Use explicit input/output instead
+- **Over-modularization:** Don't extract nodes that are only used once
+- **Deep nesting:** Keep sub-workflow call depth to 2 maximum (parent -> child)
+- **Accept all data mode:** Always define explicit inputs for maintainability
+
+## Don't Hand-Roll
+
+| Problem | Don't Build | Use Instead | Why |
+|---------|-------------|-------------|-----|
+| Workflow-to-workflow communication | Custom webhooks | Execute Sub-workflow node | Built-in, handles data passing, memory isolation |
+| Reusable logic extraction | Copy-paste nodes | Sub-workflow with trigger | Single source of truth, DRY |
+| Complex routing | Deeply nested If nodes | Switch + sub-workflows | Cleaner, testable |
+| Input validation | Manual checks in Code node | Define input fields in trigger | Type safety, documentation |
+
+**Key insight:** n8n's sub-workflow infrastructure handles all the complexity of data passing, execution context, and memory management. Using Execute Sub-workflow node instead of custom solutions provides memory isolation that prevents heap exhaustion on large operations.
+
+## Common Pitfalls
+
+### Pitfall 1: Circular Sub-workflow Calls
+**What goes wrong:** Sub-workflow A calls B which calls A, causing infinite loop
+**Why it happens:** Accidental dependency cycles during refactoring
+**How to avoid:** Document call hierarchy, keep sub-workflows single-purpose
+**Warning signs:** Workflow hangs indefinitely, memory usage spikes
+
+### Pitfall 2: Lost Context in Multi-step Flows
+**What goes wrong:** Data from early nodes not available in sub-workflow
+**Why it happens:** Sub-workflow only receives explicit inputs, not parent context
+**How to avoid:** Design complete input contracts - if sub-workflow needs data, it must be in input
+**Warning signs:** "Cannot read property of undefined" errors in sub-workflow
+
+### Pitfall 3: Credential Scope Issues
+**What goes wrong:** Sub-workflow can't access credentials defined in parent
+**Why it happens:** Credentials are workflow-scoped, not globally shared
+**How to avoid:** Configure same credentials in both parent and sub-workflows
+**Warning signs:** Authentication failures only when running as sub-workflow
+
+### Pitfall 4: Breaking Existing Callback Data
+**What goes wrong:** Inline keyboard callbacks stop working after modularization
+**Why it happens:** Callback data parsing expects specific node structure/names
+**How to avoid:** Preserve callback data format, update routing to call sub-workflows
+**Warning signs:** "Callback query expired" or routing errors after deployment
+
+### Pitfall 5: Over-extraction
+**What goes wrong:** Simple operations become 5+ sub-workflow calls, adding latency
+**Why it happens:** Applying microservice patterns too aggressively
+**How to avoid:** Only extract logic that is: (a) reused, OR (b) large/complex, OR (c) needs isolation
+**Warning signs:** Simple operations now take seconds instead of milliseconds
+
+## Code Examples
+
+### Example 1: Sub-workflow Trigger Setup
+
+```json
+{
+  "name": "Container Update",
+  "nodes": [
+    {
+      "parameters": {
+        "inputSource": "passthrough",
+        "schema": {
+          "schemaType": "fromFields",
+          "fields": [
+            { "fieldName": "containerId", "fieldType": "string" },
+            { "fieldName": "containerName", "fieldType": "string" },
+            { "fieldName": "chatId", "fieldType": "number" },
+            { "fieldName": "messageId", "fieldType": "number" },
+            { "fieldName": "responseMode", "fieldType": "string" }
+          ]
+        }
+      },
+      "name": "When executed by another workflow",
+      "type": "n8n-nodes-base.executeWorkflowTrigger",
+      "typeVersion": 1.1
+    }
+  ]
+}
+```
+
+### Example 2: Calling Sub-workflow from Parent
+
+```json
+{
+  "parameters": {
+    "source": "database",
+    "workflowId": "container-update-workflow-id",
+    "mode": "once",
+    "options": {
+      "waitForSubWorkflow": true
+    }
+  },
+  "name": "Execute Update",
+  "type": "n8n-nodes-base.executeWorkflow",
+  "typeVersion": 1.2
+}
+```
+
+### Example 3: Preparing Sub-workflow Input (Code Node)
+
+```javascript
+// In parent workflow, prepare input for sub-workflow
+const containerData = $('Match Container').item.json;
+const triggerData = $('Telegram Trigger').item.json;
+
+return {
+  json: {
+    containerId: containerData.matches[0].Id,
+    containerName: containerData.matches[0].Name,
+    chatId: triggerData.message.chat.id,
+    messageId: triggerData.message.message_id,
+    responseMode: 'text'
+  }
+};
+```
+
+### Example 4: Converting Callback Path to Use Sub-workflow
+
+```javascript
+// Instead of duplicated update logic, call sub-workflow
+const callbackData = $('Parse Callback Data').item.json;
+
+return {
+  json: {
+    containerId: callbackData.containerId,
+    containerName: callbackData.containerName,
+    chatId: callbackData.chatId,
+    messageId: callbackData.messageId,
+    responseMode: 'inline'  // Sub-workflow handles response format
+  }
+};
+```
+
+## State of the Art
+
+| Old Approach | Current Approach | When Changed | Impact |
+|--------------|------------------|--------------|--------|
+| Monolithic workflows | Modular sub-workflows | n8n 1.0+ | Better maintainability, memory efficiency |
+| Copy-paste nodes | Execute Sub-workflow node | Core feature | DRY, single source of truth |
+| Accept all data | Define using fields | 2025 best practice | Type safety, documentation |
+| Manual webhook calls | Native sub-workflow execution | Core feature | Built-in context, error handling |
+
+**Current n8n sub-workflow features (2026):**
+- Execute Sub-workflow Trigger node with schema definition
+- Sub-workflow conversion from canvas context menu (select nodes -> right-click -> Create sub-workflow)
+- Memory isolation for sub-workflows
+- Sub-workflow executions don't count toward n8n Cloud quotas
+- Wait for completion toggle for async patterns
+
+## Open Questions
+
+### 1. Sub-workflow File Organization
+**What we know:** n8n stores workflows as individual entities in its database
+**What's unclear:** Whether to export sub-workflows as separate JSON files in git or keep them n8n-internal
+**Recommendation:** Export as separate files for version control, document workflow IDs in main workflow
+
+### 2. Testing Sub-workflows
+**What we know:** Can use Manual Trigger + test data for isolated testing
+**What's unclear:** Best practice for automated testing
+**Recommendation:** Create test input nodes in each sub-workflow, manually validate before deployment
+
+### 3. Migration Strategy
+**What we know:** Can convert existing nodes to sub-workflow via context menu
+**What's unclear:** Whether to do incremental migration or big-bang refactor
+**Recommendation:** Incremental - extract one sub-workflow at a time, test, then proceed
+
+## Specific Extraction Plan
+
+Based on workflow analysis, recommended extraction order:
+
+### Priority 1: Container Update (addresses DEBT-03)
+- **Why first:** Addresses explicit tech debt requirement, duplicated in two places
+- **Lines affected:** ~1656-2400 (text path) + ~3628-4010 (callback path)
+- **Expected reduction:** ~750 lines from main workflow
+- **Risk:** Medium - update flow is critical, needs thorough testing
+
+### Priority 2: Container Simple Actions (start/stop/restart)
+- **Why second:** Simpler than update, similar pattern of text+callback duplication
+- **Expected reduction:** ~200 lines
+- **Risk:** Low - simple operations
+
+### Priority 3: Keyboard Generation
+- **Why third:** Scattered throughout, centralizing improves consistency
+- **Expected reduction:** ~300 lines
+- **Risk:** Low - UI only, easy to test
+
+### Priority 4: Batch Loop (if time permits)
+- **Why last:** Complex state management, higher risk
+- **Expected reduction:** ~400 lines
+- **Risk:** Medium-High - batch operations have edge cases
+
+## Sources
+
+### Primary (HIGH confidence)
+- [n8n Sub-workflows Documentation](https://docs.n8n.io/flow-logic/subworkflows/) - Official docs on sub-workflow concepts
+- [Execute Sub-workflow Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/) - Official node documentation
+- [Execute Sub-workflow Trigger](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflowtrigger/) - Official trigger documentation
+
+### Secondary (MEDIUM confidence)
+- [n8n Best Practices 2026](https://michaelitoback.com/n8n-workflow-best-practices/) - Community best practices guide
+- [When to Use Sub-workflows](https://community.n8n.io/t/how-to-use-sub-workflows-in-n8-when-to-use-examples/257577) - n8n Community guidance
+- [n8n Expert Best Practices](https://n8n.expert/it-automation/best-practices-designing-n8n-workflows/) - Expert recommendations
+
+### Tertiary (LOW confidence)
+- [Modularizing n8n Workflows](https://optimizesmart.com/blog/modularizing-n8n-workflows-build-smarter-workflows/) - Blog post on modularization patterns
+
+## Metadata
+
+**Confidence breakdown:**
+- Standard stack: HIGH - Based on official n8n documentation
+- Architecture patterns: MEDIUM - Based on workflow analysis + best practices
+- Pitfalls: MEDIUM - Community patterns, verified against documentation
+- Extraction plan: MEDIUM - Based on codebase analysis
+
+**Research date:** 2026-02-04
+**Valid until:** 2026-03-04 (30 days - n8n stable, patterns unlikely to change)