unraid-docker-manager/.planning/phases/10-workflow-modularization/10-RESEARCH.md

# Phase 10: Workflow Modularization - Research

**Researched:** 2026-02-04
**Domain:** n8n workflow modularization
**Confidence:** HIGH (verified against official n8n documentation)

## Summary

The current n8n workflow (`n8n-workflow.json`) is 8,485 lines with approximately 150+ nodes handling all functionality in a single monolithic workflow. This phase will break it into modular sub-workflows using n8n's built-in sub-workflow infrastructure.

n8n provides first-class support for workflow modularization through the **Execute Sub-workflow** node and **Execute Sub-workflow Trigger** node. Sub-workflows enable:
- Breaking large workflows into focused components (recommended: 5-10 nodes per sub-workflow)
- Reusing logic across multiple entry points (e.g., update flow used by text commands AND callbacks)
- Memory isolation - sub-workflow memory is released after completion
- Independent testing and debugging
- Clear data contracts between components

**Primary recommendation:** Use n8n's Execute Sub-workflow node to extract logical modules, starting with the duplicated update flow (DEBT-03) as the first extraction target since it's called from two places.

## Standard Stack

### Core n8n Nodes for Modularization

| Node | Purpose | Why Standard |
|------|---------|--------------|
| Execute Sub-workflow Trigger | Entry point for sub-workflows | Built-in n8n node, defines input contract |
| Execute Sub-workflow | Calls sub-workflows from parent | Built-in n8n node, handles data passing |
| Code Node | Data transformation between modules | Prepare inputs, format outputs |
| Switch Node | Route to different sub-workflows | Handle multiple action types |

### Input Data Modes

| Mode | When to Use |
|------|-------------|
| Define using fields | Best for typed, validated inputs (recommended) |
| Define using JSON example | When input structure is complex/nested |
| Accept all data | Legacy/migration - avoid for new sub-workflows |

**Recommendation:** Use "Define using fields" for all new sub-workflows - provides clear data contracts and automatic field population in the Execute Sub-workflow node.

## Architecture Patterns

### Recommended Module Structure

Based on workflow analysis, these logical sub-workflows are recommended:

```
Main Orchestrator (n8n-workflow.json - reduced)
├── Authentication check (inline - too small to extract)
├── Command routing (inline - just a switch node)
│
├── Sub-workflows:
│   ├── container-operations/
│   │   ├── container-action.json      # start/stop/restart a container
│   │   ├── container-update.json      # full update flow (DEBT-03 consolidation)
│   │   └── container-logs.json        # fetch and format logs
│   │
│   ├── keyboard-generation/
│   │   ├── container-list.json        # paginated container list keyboard
│   │   ├── container-submenu.json     # single container action menu
│   │   └── confirmation-dialog.json   # stop/update confirmation keyboards
│   │
│   └── batch-operations/
│       ├── batch-loop.json            # execute action on multiple containers
│       └── batch-selection.json       # selection keyboard management
```

### Pattern 1: Container Update Sub-workflow (Priority - addresses DEBT-03)

**What:** Extract the entire container update sequence into a single sub-workflow
**Why:** This logic is currently duplicated between:
1. Text command path (lines ~1656-2400)
2. Callback/inline keyboard path (lines ~3628-4010)

**Input contract (Execute Sub-workflow Trigger):**
```json
{
  "containerId": "string",
  "containerName": "string",
  "chatId": "number",
  "messageId": "number (optional - for inline updates)",
  "responseMode": "string ('text' | 'inline')"
}
```

**Output contract (returned to parent):**
```json
{
  "success": "boolean",
  "message": "string",
  "oldDigest": "string (optional)",
  "newDigest": "string (optional)"
}
```

**Flow extracted:**
1. Inspect container configuration
2. Pull latest image
3. Compare digests
4. If update needed: stop -> remove -> create -> start
5. Clean up old image
6. Return result

### Pattern 2: Keyboard Generation Sub-workflow

**What:** Extract keyboard building logic
**Why:** Keyboard generation is scattered throughout and could be centralized

**Example input:**
```json
{
  "type": "submenu",
  "containerName": "string",
  "containerState": "running|exited|paused",
  "options": {
    "showBack": true,
    "backTarget": "list:0"
  }
}
```

### Pattern 3: Data Contract Design

**Principle:** Every sub-workflow should have clearly defined inputs and outputs.

```
[Parent Workflow]
    │
    ▼ Input: { containerId, containerName, chatId, messageId }
[Execute Sub-workflow Node]
    │
    ▼ Processes via Execute Sub-workflow Trigger
[Sub-workflow: container-update]
    │
    ▼ Output: { success, message, details }
[Parent continues with result]
```

### Anti-Patterns to Avoid

- **Shared state via workflow static data:** Use explicit input/output instead
- **Over-modularization:** Don't extract nodes that are only used once
- **Deep nesting:** Keep sub-workflow call depth to 2 maximum (parent -> child)
- **Accept all data mode:** Always define explicit inputs for maintainability

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Workflow-to-workflow communication | Custom webhooks | Execute Sub-workflow node | Built-in, handles data passing, memory isolation |
| Reusable logic extraction | Copy-paste nodes | Sub-workflow with trigger | Single source of truth, DRY |
| Complex routing | Deeply nested If nodes | Switch + sub-workflows | Cleaner, testable |
| Input validation | Manual checks in Code node | Define input fields in trigger | Type safety, documentation |

**Key insight:** n8n's sub-workflow infrastructure handles all the complexity of data passing, execution context, and memory management. Using Execute Sub-workflow node instead of custom solutions provides memory isolation that prevents heap exhaustion on large operations.

## Common Pitfalls

### Pitfall 1: Circular Sub-workflow Calls
**What goes wrong:** Sub-workflow A calls B which calls A, causing infinite loop
**Why it happens:** Accidental dependency cycles during refactoring
**How to avoid:** Document call hierarchy, keep sub-workflows single-purpose
**Warning signs:** Workflow hangs indefinitely, memory usage spikes

### Pitfall 2: Lost Context in Multi-step Flows
**What goes wrong:** Data from early nodes not available in sub-workflow
**Why it happens:** Sub-workflow only receives explicit inputs, not parent context
**How to avoid:** Design complete input contracts - if sub-workflow needs data, it must be in input
**Warning signs:** "Cannot read property of undefined" errors in sub-workflow

### Pitfall 3: Credential Scope Issues
**What goes wrong:** Sub-workflow can't access credentials defined in parent
**Why it happens:** Credentials are workflow-scoped, not globally shared
**How to avoid:** Configure same credentials in both parent and sub-workflows
**Warning signs:** Authentication failures only when running as sub-workflow

### Pitfall 4: Breaking Existing Callback Data
**What goes wrong:** Inline keyboard callbacks stop working after modularization
**Why it happens:** Callback data parsing expects specific node structure/names
**How to avoid:** Preserve callback data format, update routing to call sub-workflows
**Warning signs:** "Callback query expired" or routing errors after deployment

### Pitfall 5: Over-extraction
**What goes wrong:** Simple operations become 5+ sub-workflow calls, adding latency
**Why it happens:** Applying microservice patterns too aggressively
**How to avoid:** Only extract logic that is: (a) reused, OR (b) large/complex, OR (c) needs isolation
**Warning signs:** Simple operations now take seconds instead of milliseconds

## Code Examples

### Example 1: Sub-workflow Trigger Setup

```json
{
  "name": "Container Update",
  "nodes": [
    {
      "parameters": {
        "inputSource": "passthrough",
        "schema": {
          "schemaType": "fromFields",
          "fields": [
            { "fieldName": "containerId", "fieldType": "string" },
            { "fieldName": "containerName", "fieldType": "string" },
            { "fieldName": "chatId", "fieldType": "number" },
            { "fieldName": "messageId", "fieldType": "number" },
            { "fieldName": "responseMode", "fieldType": "string" }
          ]
        }
      },
      "name": "When executed by another workflow",
      "type": "n8n-nodes-base.executeWorkflowTrigger",
      "typeVersion": 1.1
    }
  ]
}
```

### Example 2: Calling Sub-workflow from Parent

```json
{
  "parameters": {
    "source": "database",
    "workflowId": "container-update-workflow-id",
    "mode": "once",
    "options": {
      "waitForSubWorkflow": true
    }
  },
  "name": "Execute Update",
  "type": "n8n-nodes-base.executeWorkflow",
  "typeVersion": 1.2
}
```

### Example 3: Preparing Sub-workflow Input (Code Node)

```javascript
// In parent workflow, prepare input for sub-workflow
const containerData = $('Match Container').item.json;
const triggerData = $('Telegram Trigger').item.json;

return {
  json: {
    containerId: containerData.matches[0].Id,
    containerName: containerData.matches[0].Name,
    chatId: triggerData.message.chat.id,
    messageId: triggerData.message.message_id,
    responseMode: 'text'
  }
};
```

### Example 4: Converting Callback Path to Use Sub-workflow

```javascript
// Instead of duplicated update logic, call sub-workflow
const callbackData = $('Parse Callback Data').item.json;

return {
  json: {
    containerId: callbackData.containerId,
    containerName: callbackData.containerName,
    chatId: callbackData.chatId,
    messageId: callbackData.messageId,
    responseMode: 'inline'  // Sub-workflow handles response format
  }
};
```

## State of the Art

| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Monolithic workflows | Modular sub-workflows | n8n 1.0+ | Better maintainability, memory efficiency |
| Copy-paste nodes | Execute Sub-workflow node | Core feature | DRY, single source of truth |
| Accept all data | Define using fields | 2025 best practice | Type safety, documentation |
| Manual webhook calls | Native sub-workflow execution | Core feature | Built-in context, error handling |

**Current n8n sub-workflow features (2026):**
- Execute Sub-workflow Trigger node with schema definition
- Sub-workflow conversion from canvas context menu (select nodes -> right-click -> Create sub-workflow)
- Memory isolation for sub-workflows
- Sub-workflow executions don't count toward n8n Cloud quotas
- Wait for completion toggle for async patterns

## Open Questions

### 1. Sub-workflow File Organization
**What we know:** n8n stores workflows as individual entities in its database
**What's unclear:** Whether to export sub-workflows as separate JSON files in git or keep them n8n-internal
**Recommendation:** Export as separate files for version control, document workflow IDs in main workflow

### 2. Testing Sub-workflows
**What we know:** Can use Manual Trigger + test data for isolated testing
**What's unclear:** Best practice for automated testing
**Recommendation:** Create test input nodes in each sub-workflow, manually validate before deployment

### 3. Migration Strategy
**What we know:** Can convert existing nodes to sub-workflow via context menu
**What's unclear:** Whether to do incremental migration or big-bang refactor
**Recommendation:** Incremental - extract one sub-workflow at a time, test, then proceed

## Specific Extraction Plan

Based on workflow analysis, recommended extraction order:

### Priority 1: Container Update (addresses DEBT-03)
- **Why first:** Addresses explicit tech debt requirement, duplicated in two places
- **Lines affected:** ~1656-2400 (text path) + ~3628-4010 (callback path)
- **Expected reduction:** ~750 lines from main workflow
- **Risk:** Medium - update flow is critical, needs thorough testing

### Priority 2: Container Simple Actions (start/stop/restart)
- **Why second:** Simpler than update, similar pattern of text+callback duplication
- **Expected reduction:** ~200 lines
- **Risk:** Low - simple operations

### Priority 3: Keyboard Generation
- **Why third:** Scattered throughout, centralizing improves consistency
- **Expected reduction:** ~300 lines
- **Risk:** Low - UI only, easy to test

### Priority 4: Batch Loop (if time permits)
- **Why last:** Complex state management, higher risk
- **Expected reduction:** ~400 lines
- **Risk:** Medium-High - batch operations have edge cases

## Sources

### Primary (HIGH confidence)
- [n8n Sub-workflows Documentation](https://docs.n8n.io/flow-logic/subworkflows/) - Official docs on sub-workflow concepts
- [Execute Sub-workflow Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/) - Official node documentation
- [Execute Sub-workflow Trigger](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflowtrigger/) - Official trigger documentation

### Secondary (MEDIUM confidence)
- [n8n Best Practices 2026](https://michaelitoback.com/n8n-workflow-best-practices/) - Community best practices guide
- [When to Use Sub-workflows](https://community.n8n.io/t/how-to-use-sub-workflows-in-n8-when-to-use-examples/257577) - n8n Community guidance
- [n8n Expert Best Practices](https://n8n.expert/it-automation/best-practices-designing-n8n-workflows/) - Expert recommendations

### Tertiary (LOW confidence)
- [Modularizing n8n Workflows](https://optimizesmart.com/blog/modularizing-n8n-workflows-build-smarter-workflows/) - Blog post on modularization patterns

## Metadata

**Confidence breakdown:**
- Standard stack: HIGH - Based on official n8n documentation
- Architecture patterns: MEDIUM - Based on workflow analysis + best practices
- Pitfalls: MEDIUM - Community patterns, verified against documentation
- Extraction plan: MEDIUM - Based on codebase analysis

**Research date:** 2026-02-04
**Valid until:** 2026-03-04 (30 days - n8n stable, patterns unlikely to change)