ComfyUI/custom_nodes/ComfyUI-Impact-Pack/tests/wildcards

Wildcard System - Complete Test Suite

Comprehensive testing guide for the ComfyUI Impact Pack wildcard system.

---

📋 Quick Links

• Quick Start - Run tests in 5 minutes
• Test Categories - All test types
• Test Execution - How to run each test
• Troubleshooting - Common issues

---

Overview

Test Suite Structure

tests/
├── wildcards/                             # Wildcard system tests
│   ├── Unit Tests (Python)
│   │   ├── test_wildcard_lazy_loading.py      # LazyWildcardLoader class
│   │   ├── test_progressive_loading.py        # Progressive loading
│   │   ├── test_wildcard_final.py             # Final validation
│   │   └── test_lazy_load_verification.py     # Lazy load verification
│   │
│   ├── Integration Tests (Shell + API)
│   │   ├── test_progressive_ondemand.sh       # ⭐ Progressive loading (NEW)
│   │   ├── test_lazy_load_api.sh              # Lazy loading consistency
│   │   ├── test_sequential_loading.sh         # Transitive wildcards
│   │   ├── test_versatile_prompts.sh          # Feature tests
│   │   ├── test_wildcard_consistency.sh       # Consistency validation
│   │   └── test_wildcard_features.sh          # Core features
│   │
│   ├── Utility Scripts
│   │   ├── find_transitive_wildcards.sh       # Find transitive chains
│   │   ├── find_deep_transitive.py            # Deep transitive analysis
│   │   ├── verify_ondemand_mode.sh            # Verify on-demand activation
│   │   └── run_quick_test.sh                  # Quick validation
│   │
│   └── README.md (this file)
│
└── workflows/                             # Workflow test files
    ├── advanced-sampler.json
    ├── detailer-pipe-test.json
    └── ...

Test Coverage

• 11 test files (4 Python, 7 Shell)
• 100+ test scenarios
• ~95% feature coverage
• ~15 minutes total execution time

---

Quick Start

Run All Tests

cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack/tests/wildcards

# Run all shell tests
for test in test_*.sh; do
    echo "Running: $test"
    bash "$test"
done

Run Specific Test

cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack/tests/wildcards

# Progressive loading (NEW)
bash test_progressive_ondemand.sh

# Lazy loading
bash test_lazy_load_api.sh

# Sequential/transitive
bash test_sequential_loading.sh

# Versatile prompts
bash test_versatile_prompts.sh

---

Test Categories

1. Progressive On-Demand Loading Tests ⭐ NEW

Purpose: Verify wildcards are loaded progressively as accessed.

Test Files:

• test_progressive_ondemand.sh (Shell, ~2 min)
• test_progressive_loading.py (Python unit test)

What's Tested

Early Termination Size Calculation:

# Problem: 10GB scan takes 10-30 minutes
# Solution: Stop at cache limit
calculate_directory_size(path, limit=50MB)  # < 1 second

YAML Pre-loading + TXT On-Demand:

# Phase 1 (Startup): Pre-load ALL YAML files
# Reason: Keys are inside file content, not file path
load_yaml_files_only()  # colors.yaml → colors, colors/warm, colors/cold

# Phase 2 (Runtime): Load TXT files on-demand
# File path = key (e.g., "flower.txt" → "__flower__")
# No metadata scan for TXT files

Progressive Loading:

Initial:         /list/loaded → YAML keys only (e.g., colors, colors/warm, colors/cold)
After __flower__: /list/loaded → +1 TXT wildcard
After __dragon__: /list/loaded → +2-3 (TXT transitive)

⚠️ YAML Limitation: YAML wildcards are excluded from on-demand mode because wildcard keys exist inside the file content. To discover __colors/warm__, we must parse colors.yaml. Solution: Convert large YAML collections to TXT file structure for true on-demand.

New API Endpoint

GET /impact/wildcards/list/loaded:

{
  "data": ["__colors__", "__colors/warm__", "__colors/cold__", "__samples/flower__"],
  "on_demand_mode": true,
  "total_available": 0
}

Note: total_available is 0 in on-demand mode (TXT files not pre-scanned)

Progressive Example:

# Initial state (YAML pre-loaded)
curl /impact/wildcards/list/loaded
→ {"data": ["__colors__", "__colors/warm__", "__colors/cold__"], "total_available": 0}

# Access first wildcard
curl -X POST /impact/wildcards -d '{"text": "__flower__", "seed": 42}'

# Check again (TXT wildcard added)
curl /impact/wildcards/list/loaded
→ {"data": ["__colors__", "__colors/warm__", "__colors/cold__", "__samples/flower__"], "total_available": 0}

Performance Improvements

Large Dataset (10GB, 100K files):

Metric	Before	After
Startup	20-60 min	< 1 min
Memory	5-10 GB	< 100MB
Size calc	10-30 min	< 1 sec

Run Test

bash test_progressive_ondemand.sh

Expected Output:

Step 1: Initial state
  Loaded wildcards: 0

Step 2: Access __samples/flower__
  Loaded wildcards: 1
✓ PASS: Wildcard count increased

Step 3: Access __dragon__
  Loaded wildcards: 3
✓ PASS: Wildcard count increased progressively

🎉 ALL TESTS PASSED

---

2. Lazy Loading Tests

Purpose: Verify on-demand loading produces identical results to full cache mode.

Test Files:

• test_lazy_load_api.sh (Shell, ~3 min)
• test_wildcard_lazy_loading.py (Python unit test)
• test_lazy_load_verification.py (Python verification)

What's Tested

LazyWildcardLoader Class:

• Loads data only on first access
• Acts as list-like proxy
• Thread-safe with locking

Mode Detection:

• Automatic based on total size vs cache limit
• Full cache: < 50MB (default)
• On-demand: ≥ 50MB

Consistency:

• Full cache results == On-demand results
• Same seeds produce same outputs
• All wildcard features work identically

Test Scenarios

test_lazy_load_api.sh runs both modes and compares:

1. Wildcard list (before access)
2. Simple wildcard: __samples/flower__
3. Depth 3 transitive: __adnd__ creature
4. YAML wildcard: __colors__
5. Wildcard list (after access)

All results must match exactly.

Run Test

bash test_lazy_load_api.sh

Expected Output:

Testing: full_cache (limit: 100MB, port: 8190)
✓ Server started
Test 1: Get wildcard list
   Total wildcards: 1000

Testing: on_demand (limit: 1MB, port: 8191)
✓ Server started
Test 1: Get wildcard list
   Total wildcards: 1000

COMPARISON RESULTS
Test: Simple Wildcard
✓ Results MATCH

🎉 ALL TESTS PASSED
On-demand loading produces IDENTICAL results!

---

3. Sequential/Transitive Loading Tests

Purpose: Verify transitive wildcards expand correctly across multiple stages.

Test Files:

• test_sequential_loading.sh (Shell, ~5 min)
• find_transitive_wildcards.sh (Utility)

What's Tested

Transitive Expansion:

Depth 1: __samples/flower__ → rose
Depth 2: __dragon__ → __dragon/warrior__ → content
Depth 3: __adnd__ → __dragon__ → __dragon_spirit__ → content

Maximum Depth: 3 levels verified (system supports up to 100)

Test Categories

17 tests across 5 categories:

1. Depth Verification (4 tests)

   • Depth 1: Direct wildcard
   • Depth 2: One level transitive
   • Depth 3: Two levels + suffix
   • Depth 3: Maximum chain

2. Mixed Transitive (3 tests)

   • Dynamic selection of transitive
   • Multiple transitive in one prompt
   • Nested transitive in dynamic

3. Complex Scenarios (3 tests)

   • Weighted selection with transitive
   • Multi-select with transitive
   • Quantified transitive

4. Edge Cases (4 tests)

   • Compound grammar
   • Multiple wildcards, different depths
   • YAML wildcards (no transitive)
   • Transitive + YAML combination

5. On-Demand Mode (3 tests)

   • Depth 3 in on-demand
   • Complex scenario in on-demand
   • Multiple transitive in on-demand

Example: Depth 3 Chain

Files:

adnd.txt:
  __dragon__

dragon.txt:
  __dragon_spirit__

dragon_spirit.txt:
  Shrewd Hatchling
  Ancient Dragon

Usage:

__adnd__ creature
→ __dragon__ creature
→ __dragon_spirit__ creature
→ "Shrewd Hatchling creature"

Run Test

bash test_sequential_loading.sh

Expected Output:

=== Test 01: Depth 1 - Direct wildcard ===
Raw prompt: __samples/flower__
✓ All wildcards fully expanded
Final Output: rose
Status: ✅ SUCCESS

=== Test 04: Depth 3 - Maximum transitive chain ===
Raw prompt: __adnd__ creature
✓ All wildcards fully expanded
Final Output: Shrewd Hatchling creature
Status: ✅ SUCCESS

---

4. Versatile Prompts Tests

Purpose: Test all wildcard features and syntax variations.

Test Files:

• test_versatile_prompts.sh (Shell, ~2 min)
• test_wildcard_features.sh (Shell)
• test_wildcard_consistency.sh (Shell)

What's Tested

30 prompts across 10 categories:

1. Simple Wildcards (3 tests)

   • Basic substitution
   • Case insensitive (uppercase)
   • Case insensitive (mixed)

2. Dynamic Prompts (3 tests)

   • Simple: {red|green|blue} apple
   • Nested: {a|{d|e|f}|c}
   • Complex nested: {blue apple|red {cherry|berry}}

3. Selection Weights (2 tests)

   • Weighted: {5::red|4::green|7::blue} car
   • Multiple weighted: {10::beautiful|5::stunning} {3::sunset|2::sunrise}

4. Compound Grammar (3 tests)

   • Wildcard + dynamic: {pencil|apple|__flower__}
   • Complex compound: 1{girl|boy} {sitting|standing} with {__object__|item}
   • Nested compound: {big|small} {red {apple|cherry}|blue __flower__}

5. Multi-Select (4 tests)

   • Fixed count: {2$$, $$opt1|opt2|opt3|opt4}
   • Range: {2-4$$, $$opt1|opt2|opt3|opt4|opt5}
   • With separator: {3$$; $$a|b|c|d|e}
   • Short form: {-3$$, $$opt1|opt2|opt3|opt4}

6. Quantifiers (2 tests)

   • Basic: 3#__wildcard__
   • With multi-select: {2$$, $$5#__colors__}

7. Wildcard Fallback (2 tests)

   • Auto-expand: __flower__ → __*/flower__
   • Wildcard patterns: __samples/*__

8. YAML Wildcards (3 tests)

   • Simple YAML: __colors__
   • Nested YAML: __colors/warm__
   • Multiple YAML: __colors__ and __animals__

9. Transitive Wildcards (4 tests)

   • Depth 2: __dragon__
   • Depth 3: __adnd__
   • Mixed depth: __flower__ and __dragon__
   • Dynamic transitive: {__dragon__|__adnd__}

10. Real-World Scenarios (4 tests)

    • Portrait prompt
    • Landscape prompt
    • Fantasy prompt
    • Abstract art prompt

Example Tests

Test 04: Simple Dynamic Prompt:

Raw: {red|green|blue} apple
Seed: 100
Result: "red apple" (deterministic)

Test 09: Wildcard + Dynamic:

Raw: 1girl holding {blue pencil|red apple|colorful __samples/flower__}
Seed: 100
Result: "1girl holding colorful chrysanthemum"

Test 18: Multi-Select Range:

Raw: {2-4$$, $$happy|sad|angry|excited|calm}
Seed: 100
Result: "happy, sad, angry" (2-4 emotions selected)

Run Test

bash test_versatile_prompts.sh

Expected Output:

========================================
Test 01: Basic Wildcard
========================================
Raw: __samples/flower__
Result: chrysanthemum
Status: ✅ PASS

========================================
Test 04: Simple Dynamic Prompt
========================================
Raw: {red|green|blue} apple
Result: red apple
Status: ✅ PASS

Total: 30 tests
Passed: 30
Failed: 0

---

Test Execution

Prerequisites

Required:

• ComfyUI installed
• Impact Pack installed
• Python 3.8+
• Bash shell
• curl (for API tests)

Optional:

• jq (for JSON parsing)
• git (for version control)

Environment Setup

1. Configure Impact Pack:

cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack

# Create or edit config
cat > impact-pack.ini << EOF
[default]
dependency_version = 24
wildcard_cache_limit_mb = 50
custom_wildcards = $(pwd)/custom_wildcards
disable_gpu_opencv = True
EOF

2. Prepare Wildcards:

# Check wildcard files exist
ls wildcards/*.txt wildcards/*.yaml
ls custom_wildcards/*.txt

Running Tests

Unit Tests (Python)

Standalone (no server required):

python3 test_wildcard_lazy_loading.py
python3 test_progressive_loading.py

Note: Requires ComfyUI environment or will show import errors.

Integration Tests (Shell)

Manual Server Start:

# Terminal 1: Start server
cd /path/to/ComfyUI
bash run.sh --listen 127.0.0.1 --port 8188

# Terminal 2: Run tests
cd custom_nodes/comfyui-impact-pack/tests
bash test_versatile_prompts.sh

Automated (tests start/stop server):

# Each test manages its own server
bash test_progressive_ondemand.sh  # Port 8195
bash test_lazy_load_api.sh         # Ports 8190-8191
bash test_sequential_loading.sh    # Port 8193

Test Timing

Test	Duration	Server	Ports
test_progressive_ondemand.sh	~2 min	Auto	8195
test_lazy_load_api.sh	~3 min	Auto	8190-8191
test_sequential_loading.sh	~5 min	Auto	8193
test_versatile_prompts.sh	~2 min	Manual	8188
test_wildcard_consistency.sh	~1 min	Manual	8188
Python unit tests	< 5 sec	No	N/A

Logs

Server Logs:

/tmp/progressive_test.log
/tmp/comfyui_full_cache.log
/tmp/comfyui_on_demand.log
/tmp/sequential_test.log

Check Logs:

# View recent wildcard logs
tail -50 /tmp/progressive_test.log | grep -i wildcard

# Find errors
grep -i "error\|fail" /tmp/*.log

# Check mode activation
grep -i "mode" /tmp/progressive_test.log

---

Expected Results

Success Criteria

Progressive Loading

• ✅ /list/loaded starts at 0 (or low count)
• ✅ /list/loaded increases after each unique wildcard
• ✅ /list/loaded unchanged on cache hits
• ✅ Transitive wildcards load multiple entries
• ✅ Final results identical to full cache mode

Lazy Loading

• ✅ Full cache results == On-demand results (all tests)
• ✅ Mode detection correct (based on size vs limit)
• ✅ LazyWildcardLoader loads only on access
• ✅ All API endpoints return consistent data

Sequential Loading

• ✅ Depth 1-3 expand correctly
• ✅ Complex scenarios work (weighted, multi-select, etc.)
• ✅ On-demand mode matches full cache
• ✅ No infinite loops (max 100 iterations)

Versatile Prompts

• ✅ All 30 test prompts process successfully
• ✅ Deterministic (same seed → same result)
• ✅ No syntax errors
• ✅ Proper probability distribution

Sample Output

Progressive Loading Success:

========================================
Progressive Loading Verification
========================================

Step 1: Initial state
  On-demand mode: True
  Total available: 1000
  Loaded wildcards: 0

Step 2: Access __samples/flower__
  Result: rose
  Loaded wildcards: 1
✓ PASS

Step 3: Access __dragon__
  Result: ancient dragon
  Loaded wildcards: 3
✓ PASS

🎉 ALL TESTS PASSED
Progressive on-demand loading verified!

Lazy Loading Success:

========================================
COMPARISON RESULTS
========================================

Test: Wildcard List (before)
✓ Results MATCH

Test: Simple Wildcard
✓ Results MATCH

Test: Depth 3 Transitive
✓ Results MATCH

🎉 ALL TESTS PASSED
On-demand produces IDENTICAL results!

---

Troubleshooting

Common Issues

1. Server Fails to Start

Symptoms:

✗ Server failed to start
curl: (7) Failed to connect

Solutions:

# Check if port in use
lsof -i :8188
netstat -tlnp | grep 8188

# Kill existing processes
pkill -f "python.*main.py"

# Increase startup wait time
# In test script: sleep 15 → sleep 30

2. Module Not Found (Python)

Symptoms:

ModuleNotFoundError: No module named 'modules'

Solutions:

# Option 1: Run from ComfyUI directory
cd /path/to/ComfyUI
python3 custom_nodes/comfyui-impact-pack/tests/test_progressive_loading.py

# Option 2: Add to PYTHONPATH
export PYTHONPATH=/path/to/ComfyUI/custom_nodes/comfyui-impact-pack:$PYTHONPATH
python3 test_progressive_loading.py

3. On-Demand Mode Not Activating

Symptoms:

Using full cache mode.

Check:

# View total size
grep "Wildcard total size" /tmp/progressive_test.log

# Check cache limit
grep "cache_limit_mb" impact-pack.ini

Solutions:

# Force on-demand mode
cat > impact-pack.ini << EOF
[default]
wildcard_cache_limit_mb = 0.5
EOF

4. Tests Timeout

Symptoms:

Waiting for server startup...
✗ Server failed to start

Solutions:

# Check system resources
free -h
df -h

# View server logs
tail -100 /tmp/progressive_test.log

# Manually test server
cd /path/to/ComfyUI
bash run.sh --port 8195

# Increase timeout in test
# sleep 15 → sleep 60

5. Results Don't Match

Symptoms:

✗ Results DIFFER

Debug:

# Compare results
diff /tmp/result_full_cache_simple.json /tmp/result_on_demand_simple.json

# Check seeds are same
grep "seed" /tmp/result_*.json

# Verify same wildcard files used
ls -la wildcards/samples/flower.txt

File Bug Report:

• Wildcard text
• Seed value
• Full cache result
• On-demand result
• Server logs

6. Slow Performance

Symptoms:

• Tests take much longer than expected
• Server startup > 2 minutes

Check:

# Wildcard size
du -sh wildcards/

# Disk I/O
iostat -x 1 5

# System resources
top

Solutions:

• Use SSD (not HDD)
• Reduce wildcard size
• Increase cache limit (use full cache mode)
• Close other applications

---

Performance Benchmarks

Expected Performance

Small Dataset (< 50MB):

Mode: Full cache
Startup: < 10 seconds
Memory: ~50MB
First access: Instant

Medium Dataset (50MB - 1GB):

Mode: On-demand
Startup: < 30 seconds
Memory: < 200MB initial
First access: 10-50ms per wildcard

Large Dataset (10GB+):

Mode: On-demand
Startup: < 1 minute
Memory: < 100MB initial
First access: 10-50ms per wildcard
Memory growth: Progressive

Optimization Tips

For Faster Tests:

1. Use smaller wildcard dataset
2. Run specific tests (not all)
3. Use manual server (keep running)
4. Skip sleep times (if server already running)

For Large Datasets:

1. Verify on-demand mode activates
2. Monitor /list/loaded to track memory
3. Use SSD for file storage
4. Organize wildcards into subdirectories

---

Contributing

Adding New Tests

1. Create Test File:

touch tests/test_new_feature.sh
chmod +x tests/test_new_feature.sh

2. Test Template:

#!/bin/bash
# Test: New Feature
# Purpose: Verify new feature works correctly

set -e

PORT=8XXX
IMPACT_DIR="/path/to/comfyui-impact-pack"

# Setup config
cat > impact-pack.ini << EOF
[default]
wildcard_cache_limit_mb = 50
EOF

# Start server
cd /path/to/ComfyUI
bash run.sh --port $PORT > /tmp/test_new.log 2>&1 &
sleep 15

# Test
RESULT=$(curl -s http://127.0.0.1:$PORT/impact/wildcards/list)

# Validate
if [ "$RESULT" = "expected" ]; then
    echo "✅ PASS"
    exit 0
else
    echo "❌ FAIL"
    exit 1
fi

3. Update Documentation:

• Add test description to this README
• Update test count
• Add to appropriate category

Testing Guidelines

Test Structure:

1. Clear purpose statement
2. Setup (config, wildcards)
3. Execution (API calls, processing)
4. Validation (assertions, comparisons)
5. Cleanup (kill servers, restore config)

Good Practices:

• Use unique port numbers
• Clean up background processes
• Provide clear success/failure messages
• Log to /tmp/ for debugging
• Use deterministic seeds
• Test both modes (full cache + on-demand)

---

Reference

Test Files Quick Reference

# Progressive loading
test_progressive_ondemand.sh          # Integration test
test_progressive_loading.py           # Unit test

# Lazy loading
test_lazy_load_api.sh                 # Integration test
test_wildcard_lazy_loading.py         # Unit test

# Sequential/transitive
test_sequential_loading.sh            # Integration test
find_transitive_wildcards.sh          # Utility

# Features
test_versatile_prompts.sh             # Comprehensive features
test_wildcard_features.sh             # Core features
test_wildcard_consistency.sh          # Consistency

# Validation
test_wildcard_final.py                # Final validation
test_lazy_load_verification.py        # Lazy load verification

Documentation

• System Overview: ../docs/WILDCARD_SYSTEM_OVERVIEW.md
• Testing Guide: ../docs/WILDCARD_TESTING_GUIDE.md

API Endpoints

GET  /impact/wildcards/list           # All available wildcards
GET  /impact/wildcards/list/loaded    # Actually loaded (progressive)
POST /impact/wildcards                # Process wildcard text
GET  /impact/wildcards/refresh        # Reload all wildcards

---

Last Updated: 2024-11-17 Total Tests: 11 files, 100+ scenarios Coverage: ~95% of wildcard features