Add custom nodes, Civitai loras (LFS), and vast.ai setup script
Some checks failed
Python Linting / Run Ruff (push) Has been cancelled
Python Linting / Run Pylint (push) Has been cancelled
Full Comfy CI Workflow Runs / test-stable (12.1, , linux, 3.10, [self-hosted Linux], stable) (push) Has been cancelled
Full Comfy CI Workflow Runs / test-stable (12.1, , linux, 3.11, [self-hosted Linux], stable) (push) Has been cancelled
Full Comfy CI Workflow Runs / test-stable (12.1, , linux, 3.12, [self-hosted Linux], stable) (push) Has been cancelled
Full Comfy CI Workflow Runs / test-unix-nightly (12.1, , linux, 3.11, [self-hosted Linux], nightly) (push) Has been cancelled
Execution Tests / test (macos-latest) (push) Has been cancelled
Execution Tests / test (ubuntu-latest) (push) Has been cancelled
Execution Tests / test (windows-latest) (push) Has been cancelled
Test server launches without errors / test (push) Has been cancelled
Unit Tests / test (macos-latest) (push) Has been cancelled
Unit Tests / test (ubuntu-latest) (push) Has been cancelled
Unit Tests / test (windows-2022) (push) Has been cancelled
Some checks failed
Python Linting / Run Ruff (push) Has been cancelled
Python Linting / Run Pylint (push) Has been cancelled
Full Comfy CI Workflow Runs / test-stable (12.1, , linux, 3.10, [self-hosted Linux], stable) (push) Has been cancelled
Full Comfy CI Workflow Runs / test-stable (12.1, , linux, 3.11, [self-hosted Linux], stable) (push) Has been cancelled
Full Comfy CI Workflow Runs / test-stable (12.1, , linux, 3.12, [self-hosted Linux], stable) (push) Has been cancelled
Full Comfy CI Workflow Runs / test-unix-nightly (12.1, , linux, 3.11, [self-hosted Linux], nightly) (push) Has been cancelled
Execution Tests / test (macos-latest) (push) Has been cancelled
Execution Tests / test (ubuntu-latest) (push) Has been cancelled
Execution Tests / test (windows-latest) (push) Has been cancelled
Test server launches without errors / test (push) Has been cancelled
Unit Tests / test (macos-latest) (push) Has been cancelled
Unit Tests / test (ubuntu-latest) (push) Has been cancelled
Unit Tests / test (windows-2022) (push) Has been cancelled
Includes 30 custom nodes committed directly, 7 Civitai-exclusive loras stored via Git LFS, and a setup script that installs all dependencies and downloads HuggingFace-hosted models on vast.ai. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
961
custom_nodes/ComfyUI-Impact-Pack/tests/wildcards/README.md
Normal file
961
custom_nodes/ComfyUI-Impact-Pack/tests/wildcards/README.md
Normal file
@@ -0,0 +1,961 @@
|
||||
# Wildcard System - Complete Test Suite
|
||||
|
||||
Comprehensive testing guide for the ComfyUI Impact Pack wildcard system.
|
||||
|
||||
---
|
||||
|
||||
## 📋 Quick Links
|
||||
|
||||
- **[Quick Start](#quick-start)** - Run tests in 5 minutes
|
||||
- **[Test Categories](#test-categories)** - All test types
|
||||
- **[Test Execution](#test-execution)** - How to run each test
|
||||
- **[Troubleshooting](#troubleshooting)** - Common issues
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
### Test Suite Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── wildcards/ # Wildcard system tests
|
||||
│ ├── Unit Tests (Python)
|
||||
│ │ ├── test_wildcard_lazy_loading.py # LazyWildcardLoader class
|
||||
│ │ ├── test_progressive_loading.py # Progressive loading
|
||||
│ │ ├── test_wildcard_final.py # Final validation
|
||||
│ │ └── test_lazy_load_verification.py # Lazy load verification
|
||||
│ │
|
||||
│ ├── Integration Tests (Shell + API)
|
||||
│ │ ├── test_progressive_ondemand.sh # ⭐ Progressive loading (NEW)
|
||||
│ │ ├── test_lazy_load_api.sh # Lazy loading consistency
|
||||
│ │ ├── test_sequential_loading.sh # Transitive wildcards
|
||||
│ │ ├── test_versatile_prompts.sh # Feature tests
|
||||
│ │ ├── test_wildcard_consistency.sh # Consistency validation
|
||||
│ │ └── test_wildcard_features.sh # Core features
|
||||
│ │
|
||||
│ ├── Utility Scripts
|
||||
│ │ ├── find_transitive_wildcards.sh # Find transitive chains
|
||||
│ │ ├── find_deep_transitive.py # Deep transitive analysis
|
||||
│ │ ├── verify_ondemand_mode.sh # Verify on-demand activation
|
||||
│ │ └── run_quick_test.sh # Quick validation
|
||||
│ │
|
||||
│ └── README.md (this file)
|
||||
│
|
||||
└── workflows/ # Workflow test files
|
||||
├── advanced-sampler.json
|
||||
├── detailer-pipe-test.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Test Coverage
|
||||
|
||||
- **11 test files** (4 Python, 7 Shell)
|
||||
- **100+ test scenarios**
|
||||
- **~95% feature coverage**
|
||||
- **~15 minutes** total execution time
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Run All Tests
|
||||
|
||||
```bash
|
||||
cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack/tests/wildcards
|
||||
|
||||
# Run all shell tests
|
||||
for test in test_*.sh; do
|
||||
echo "Running: $test"
|
||||
bash "$test"
|
||||
done
|
||||
```
|
||||
|
||||
### Run Specific Test
|
||||
|
||||
```bash
|
||||
cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack/tests/wildcards
|
||||
|
||||
# Progressive loading (NEW)
|
||||
bash test_progressive_ondemand.sh
|
||||
|
||||
# Lazy loading
|
||||
bash test_lazy_load_api.sh
|
||||
|
||||
# Sequential/transitive
|
||||
bash test_sequential_loading.sh
|
||||
|
||||
# Versatile prompts
|
||||
bash test_versatile_prompts.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Progressive On-Demand Loading Tests ⭐ NEW
|
||||
|
||||
**Purpose**: Verify wildcards are loaded progressively as accessed.
|
||||
|
||||
**Test Files**:
|
||||
- `test_progressive_ondemand.sh` (Shell, ~2 min)
|
||||
- `test_progressive_loading.py` (Python unit test)
|
||||
|
||||
#### What's Tested
|
||||
|
||||
**Early Termination Size Calculation**:
|
||||
```python
|
||||
# Problem: 10GB scan takes 10-30 minutes
|
||||
# Solution: Stop at cache limit
|
||||
calculate_directory_size(path, limit=50MB) # < 1 second
|
||||
```
|
||||
|
||||
**YAML Pre-loading + TXT On-Demand**:
|
||||
```python
|
||||
# Phase 1 (Startup): Pre-load ALL YAML files
|
||||
# Reason: Keys are inside file content, not file path
|
||||
load_yaml_files_only() # colors.yaml → colors, colors/warm, colors/cold
|
||||
|
||||
# Phase 2 (Runtime): Load TXT files on-demand
|
||||
# File path = key (e.g., "flower.txt" → "__flower__")
|
||||
# No metadata scan for TXT files
|
||||
```
|
||||
|
||||
**Progressive Loading**:
|
||||
```
|
||||
Initial: /list/loaded → YAML keys only (e.g., colors, colors/warm, colors/cold)
|
||||
After __flower__: /list/loaded → +1 TXT wildcard
|
||||
After __dragon__: /list/loaded → +2-3 (TXT transitive)
|
||||
```
|
||||
|
||||
**⚠️ YAML Limitation**:
|
||||
YAML wildcards are excluded from on-demand mode because wildcard keys exist
|
||||
inside the file content. To discover `__colors/warm__`, we must parse `colors.yaml`.
|
||||
Solution: Convert large YAML collections to TXT file structure for true on-demand.
|
||||
|
||||
#### New API Endpoint
|
||||
|
||||
**`GET /impact/wildcards/list/loaded`**:
|
||||
```json
|
||||
{
|
||||
"data": ["__colors__", "__colors/warm__", "__colors/cold__", "__samples/flower__"],
|
||||
"on_demand_mode": true,
|
||||
"total_available": 0
|
||||
}
|
||||
```
|
||||
|
||||
Note: `total_available` is 0 in on-demand mode (TXT files not pre-scanned)
|
||||
|
||||
**Progressive Example**:
|
||||
```bash
|
||||
# Initial state (YAML pre-loaded)
|
||||
curl /impact/wildcards/list/loaded
|
||||
→ {"data": ["__colors__", "__colors/warm__", "__colors/cold__"], "total_available": 0}
|
||||
|
||||
# Access first wildcard
|
||||
curl -X POST /impact/wildcards -d '{"text": "__flower__", "seed": 42}'
|
||||
|
||||
# Check again (TXT wildcard added)
|
||||
curl /impact/wildcards/list/loaded
|
||||
→ {"data": ["__colors__", "__colors/warm__", "__colors/cold__", "__samples/flower__"], "total_available": 0}
|
||||
```
|
||||
|
||||
#### Performance Improvements
|
||||
|
||||
**Large Dataset (10GB, 100K files)**:
|
||||
|
||||
| Metric | Before | After |
|
||||
|--------|--------|-------|
|
||||
| **Startup** | 20-60 min | **< 1 min** |
|
||||
| **Memory** | 5-10 GB | **< 100MB** |
|
||||
| **Size calc** | 10-30 min | **< 1 sec** |
|
||||
|
||||
#### Run Test
|
||||
|
||||
```bash
|
||||
bash test_progressive_ondemand.sh
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
```
|
||||
Step 1: Initial state
|
||||
Loaded wildcards: 0
|
||||
|
||||
Step 2: Access __samples/flower__
|
||||
Loaded wildcards: 1
|
||||
✓ PASS: Wildcard count increased
|
||||
|
||||
Step 3: Access __dragon__
|
||||
Loaded wildcards: 3
|
||||
✓ PASS: Wildcard count increased progressively
|
||||
|
||||
🎉 ALL TESTS PASSED
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Lazy Loading Tests
|
||||
|
||||
**Purpose**: Verify on-demand loading produces identical results to full cache mode.
|
||||
|
||||
**Test Files**:
|
||||
- `test_lazy_load_api.sh` (Shell, ~3 min)
|
||||
- `test_wildcard_lazy_loading.py` (Python unit test)
|
||||
- `test_lazy_load_verification.py` (Python verification)
|
||||
|
||||
#### What's Tested
|
||||
|
||||
**LazyWildcardLoader Class**:
|
||||
- Loads data only on first access
|
||||
- Acts as list-like proxy
|
||||
- Thread-safe with locking
|
||||
|
||||
**Mode Detection**:
|
||||
- Automatic based on total size vs cache limit
|
||||
- Full cache: < 50MB (default)
|
||||
- On-demand: ≥ 50MB
|
||||
|
||||
**Consistency**:
|
||||
- Full cache results == On-demand results
|
||||
- Same seeds produce same outputs
|
||||
- All wildcard features work identically
|
||||
|
||||
#### Test Scenarios
|
||||
|
||||
**test_lazy_load_api.sh** runs both modes and compares:
|
||||
|
||||
1. **Wildcard list** (before access)
|
||||
2. **Simple wildcard**: `__samples/flower__`
|
||||
3. **Depth 3 transitive**: `__adnd__ creature`
|
||||
4. **YAML wildcard**: `__colors__`
|
||||
5. **Wildcard list** (after access)
|
||||
|
||||
**All results must match exactly**.
|
||||
|
||||
#### Run Test
|
||||
|
||||
```bash
|
||||
bash test_lazy_load_api.sh
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
```
|
||||
Testing: full_cache (limit: 100MB, port: 8190)
|
||||
✓ Server started
|
||||
Test 1: Get wildcard list
|
||||
Total wildcards: 1000
|
||||
|
||||
Testing: on_demand (limit: 1MB, port: 8191)
|
||||
✓ Server started
|
||||
Test 1: Get wildcard list
|
||||
Total wildcards: 1000
|
||||
|
||||
COMPARISON RESULTS
|
||||
Test: Simple Wildcard
|
||||
✓ Results MATCH
|
||||
|
||||
🎉 ALL TESTS PASSED
|
||||
On-demand loading produces IDENTICAL results!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Sequential/Transitive Loading Tests
|
||||
|
||||
**Purpose**: Verify transitive wildcards expand correctly across multiple stages.
|
||||
|
||||
**Test Files**:
|
||||
- `test_sequential_loading.sh` (Shell, ~5 min)
|
||||
- `find_transitive_wildcards.sh` (Utility)
|
||||
|
||||
#### What's Tested
|
||||
|
||||
**Transitive Expansion**:
|
||||
```
|
||||
Depth 1: __samples/flower__ → rose
|
||||
Depth 2: __dragon__ → __dragon/warrior__ → content
|
||||
Depth 3: __adnd__ → __dragon__ → __dragon_spirit__ → content
|
||||
```
|
||||
|
||||
**Maximum Depth**: 3 levels verified (system supports up to 100)
|
||||
|
||||
#### Test Categories
|
||||
|
||||
**17 tests across 5 categories**:
|
||||
|
||||
1. **Depth Verification** (4 tests)
|
||||
- Depth 1: Direct wildcard
|
||||
- Depth 2: One level transitive
|
||||
- Depth 3: Two levels + suffix
|
||||
- Depth 3: Maximum chain
|
||||
|
||||
2. **Mixed Transitive** (3 tests)
|
||||
- Dynamic selection of transitive
|
||||
- Multiple transitive in one prompt
|
||||
- Nested transitive in dynamic
|
||||
|
||||
3. **Complex Scenarios** (3 tests)
|
||||
- Weighted selection with transitive
|
||||
- Multi-select with transitive
|
||||
- Quantified transitive
|
||||
|
||||
4. **Edge Cases** (4 tests)
|
||||
- Compound grammar
|
||||
- Multiple wildcards, different depths
|
||||
- YAML wildcards (no transitive)
|
||||
- Transitive + YAML combination
|
||||
|
||||
5. **On-Demand Mode** (3 tests)
|
||||
- Depth 3 in on-demand
|
||||
- Complex scenario in on-demand
|
||||
- Multiple transitive in on-demand
|
||||
|
||||
#### Example: Depth 3 Chain
|
||||
|
||||
**Files**:
|
||||
```
|
||||
adnd.txt:
|
||||
__dragon__
|
||||
|
||||
dragon.txt:
|
||||
__dragon_spirit__
|
||||
|
||||
dragon_spirit.txt:
|
||||
Shrewd Hatchling
|
||||
Ancient Dragon
|
||||
```
|
||||
|
||||
**Usage**:
|
||||
```
|
||||
__adnd__ creature
|
||||
→ __dragon__ creature
|
||||
→ __dragon_spirit__ creature
|
||||
→ "Shrewd Hatchling creature"
|
||||
```
|
||||
|
||||
#### Run Test
|
||||
|
||||
```bash
|
||||
bash test_sequential_loading.sh
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
```
|
||||
=== Test 01: Depth 1 - Direct wildcard ===
|
||||
Raw prompt: __samples/flower__
|
||||
✓ All wildcards fully expanded
|
||||
Final Output: rose
|
||||
Status: ✅ SUCCESS
|
||||
|
||||
=== Test 04: Depth 3 - Maximum transitive chain ===
|
||||
Raw prompt: __adnd__ creature
|
||||
✓ All wildcards fully expanded
|
||||
Final Output: Shrewd Hatchling creature
|
||||
Status: ✅ SUCCESS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Versatile Prompts Tests
|
||||
|
||||
**Purpose**: Test all wildcard features and syntax variations.
|
||||
|
||||
**Test Files**:
|
||||
- `test_versatile_prompts.sh` (Shell, ~2 min)
|
||||
- `test_wildcard_features.sh` (Shell)
|
||||
- `test_wildcard_consistency.sh` (Shell)
|
||||
|
||||
#### What's Tested
|
||||
|
||||
**30 prompts across 10 categories**:
|
||||
|
||||
1. **Simple Wildcards** (3 tests)
|
||||
- Basic substitution
|
||||
- Case insensitive (uppercase)
|
||||
- Case insensitive (mixed)
|
||||
|
||||
2. **Dynamic Prompts** (3 tests)
|
||||
- Simple: `{red|green|blue} apple`
|
||||
- Nested: `{a|{d|e|f}|c}`
|
||||
- Complex nested: `{blue apple|red {cherry|berry}}`
|
||||
|
||||
3. **Selection Weights** (2 tests)
|
||||
- Weighted: `{5::red|4::green|7::blue} car`
|
||||
- Multiple weighted: `{10::beautiful|5::stunning} {3::sunset|2::sunrise}`
|
||||
|
||||
4. **Compound Grammar** (3 tests)
|
||||
- Wildcard + dynamic: `{pencil|apple|__flower__}`
|
||||
- Complex compound: `1{girl|boy} {sitting|standing} with {__object__|item}`
|
||||
- Nested compound: `{big|small} {red {apple|cherry}|blue __flower__}`
|
||||
|
||||
5. **Multi-Select** (4 tests)
|
||||
- Fixed count: `{2$$, $$opt1|opt2|opt3|opt4}`
|
||||
- Range: `{2-4$$, $$opt1|opt2|opt3|opt4|opt5}`
|
||||
- With separator: `{3$$; $$a|b|c|d|e}`
|
||||
- Short form: `{-3$$, $$opt1|opt2|opt3|opt4}`
|
||||
|
||||
6. **Quantifiers** (2 tests)
|
||||
- Basic: `3#__wildcard__`
|
||||
- With multi-select: `{2$$, $$5#__colors__}`
|
||||
|
||||
7. **Wildcard Fallback** (2 tests)
|
||||
- Auto-expand: `__flower__` → `__*/flower__`
|
||||
- Wildcard patterns: `__samples/*__`
|
||||
|
||||
8. **YAML Wildcards** (3 tests)
|
||||
- Simple YAML: `__colors__`
|
||||
- Nested YAML: `__colors/warm__`
|
||||
- Multiple YAML: `__colors__ and __animals__`
|
||||
|
||||
9. **Transitive Wildcards** (4 tests)
|
||||
- Depth 2: `__dragon__`
|
||||
- Depth 3: `__adnd__`
|
||||
- Mixed depth: `__flower__ and __dragon__`
|
||||
- Dynamic transitive: `{__dragon__|__adnd__}`
|
||||
|
||||
10. **Real-World Scenarios** (4 tests)
|
||||
- Portrait prompt
|
||||
- Landscape prompt
|
||||
- Fantasy prompt
|
||||
- Abstract art prompt
|
||||
|
||||
#### Example Tests
|
||||
|
||||
**Test 04: Simple Dynamic Prompt**:
|
||||
```
|
||||
Raw: {red|green|blue} apple
|
||||
Seed: 100
|
||||
Result: "red apple" (deterministic)
|
||||
```
|
||||
|
||||
**Test 09: Wildcard + Dynamic**:
|
||||
```
|
||||
Raw: 1girl holding {blue pencil|red apple|colorful __samples/flower__}
|
||||
Seed: 100
|
||||
Result: "1girl holding colorful chrysanthemum"
|
||||
```
|
||||
|
||||
**Test 18: Multi-Select Range**:
|
||||
```
|
||||
Raw: {2-4$$, $$happy|sad|angry|excited|calm}
|
||||
Seed: 100
|
||||
Result: "happy, sad, angry" (2-4 emotions selected)
|
||||
```
|
||||
|
||||
#### Run Test
|
||||
|
||||
```bash
|
||||
bash test_versatile_prompts.sh
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
```
|
||||
========================================
|
||||
Test 01: Basic Wildcard
|
||||
========================================
|
||||
Raw: __samples/flower__
|
||||
Result: chrysanthemum
|
||||
Status: ✅ PASS
|
||||
|
||||
========================================
|
||||
Test 04: Simple Dynamic Prompt
|
||||
========================================
|
||||
Raw: {red|green|blue} apple
|
||||
Result: red apple
|
||||
Status: ✅ PASS
|
||||
|
||||
Total: 30 tests
|
||||
Passed: 30
|
||||
Failed: 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Execution
|
||||
|
||||
### Prerequisites
|
||||
|
||||
**Required**:
|
||||
- ComfyUI installed
|
||||
- Impact Pack installed
|
||||
- Python 3.8+
|
||||
- Bash shell
|
||||
- curl (for API tests)
|
||||
|
||||
**Optional**:
|
||||
- jq (for JSON parsing)
|
||||
- git (for version control)
|
||||
|
||||
### Environment Setup
|
||||
|
||||
**1. Configure Impact Pack**:
|
||||
```bash
|
||||
cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack
|
||||
|
||||
# Create or edit config
|
||||
cat > impact-pack.ini << EOF
|
||||
[default]
|
||||
dependency_version = 24
|
||||
wildcard_cache_limit_mb = 50
|
||||
custom_wildcards = $(pwd)/custom_wildcards
|
||||
disable_gpu_opencv = True
|
||||
EOF
|
||||
```
|
||||
|
||||
**2. Prepare Wildcards**:
|
||||
```bash
|
||||
# Check wildcard files exist
|
||||
ls wildcards/*.txt wildcards/*.yaml
|
||||
ls custom_wildcards/*.txt
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
#### Unit Tests (Python)
|
||||
|
||||
**Standalone** (no server required):
|
||||
```bash
|
||||
python3 test_wildcard_lazy_loading.py
|
||||
python3 test_progressive_loading.py
|
||||
```
|
||||
|
||||
**Note**: Requires ComfyUI environment or will show import errors.
|
||||
|
||||
#### Integration Tests (Shell)
|
||||
|
||||
**Manual Server Start**:
|
||||
```bash
|
||||
# Terminal 1: Start server
|
||||
cd /path/to/ComfyUI
|
||||
bash run.sh --listen 127.0.0.1 --port 8188
|
||||
|
||||
# Terminal 2: Run tests
|
||||
cd custom_nodes/comfyui-impact-pack/tests
|
||||
bash test_versatile_prompts.sh
|
||||
```
|
||||
|
||||
**Automated** (tests start/stop server):
|
||||
```bash
|
||||
# Each test manages its own server
|
||||
bash test_progressive_ondemand.sh # Port 8195
|
||||
bash test_lazy_load_api.sh # Ports 8190-8191
|
||||
bash test_sequential_loading.sh # Port 8193
|
||||
```
|
||||
|
||||
### Test Timing
|
||||
|
||||
| Test | Duration | Server | Ports |
|
||||
|------|----------|--------|-------|
|
||||
| `test_progressive_ondemand.sh` | ~2 min | Auto | 8195 |
|
||||
| `test_lazy_load_api.sh` | ~3 min | Auto | 8190-8191 |
|
||||
| `test_sequential_loading.sh` | ~5 min | Auto | 8193 |
|
||||
| `test_versatile_prompts.sh` | ~2 min | Manual | 8188 |
|
||||
| `test_wildcard_consistency.sh` | ~1 min | Manual | 8188 |
|
||||
| Python unit tests | < 5 sec | No | N/A |
|
||||
|
||||
### Logs
|
||||
|
||||
**Server Logs**:
|
||||
```bash
|
||||
/tmp/progressive_test.log
|
||||
/tmp/comfyui_full_cache.log
|
||||
/tmp/comfyui_on_demand.log
|
||||
/tmp/sequential_test.log
|
||||
```
|
||||
|
||||
**Check Logs**:
|
||||
```bash
|
||||
# View recent wildcard logs
|
||||
tail -50 /tmp/progressive_test.log | grep -i wildcard
|
||||
|
||||
# Find errors
|
||||
grep -i "error\|fail" /tmp/*.log
|
||||
|
||||
# Check mode activation
|
||||
grep -i "mode" /tmp/progressive_test.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Success Criteria
|
||||
|
||||
#### Progressive Loading
|
||||
- ✅ `/list/loaded` starts at 0 (or low count)
|
||||
- ✅ `/list/loaded` increases after each unique wildcard
|
||||
- ✅ `/list/loaded` unchanged on cache hits
|
||||
- ✅ Transitive wildcards load multiple entries
|
||||
- ✅ Final results identical to full cache mode
|
||||
|
||||
#### Lazy Loading
|
||||
- ✅ Full cache results == On-demand results (all tests)
|
||||
- ✅ Mode detection correct (based on size vs limit)
|
||||
- ✅ LazyWildcardLoader loads only on access
|
||||
- ✅ All API endpoints return consistent data
|
||||
|
||||
#### Sequential Loading
|
||||
- ✅ Depth 1-3 expand correctly
|
||||
- ✅ Complex scenarios work (weighted, multi-select, etc.)
|
||||
- ✅ On-demand mode matches full cache
|
||||
- ✅ No infinite loops (max 100 iterations)
|
||||
|
||||
#### Versatile Prompts
|
||||
- ✅ All 30 test prompts process successfully
|
||||
- ✅ Deterministic (same seed → same result)
|
||||
- ✅ No syntax errors
|
||||
- ✅ Proper probability distribution
|
||||
|
||||
### Sample Output
|
||||
|
||||
**Progressive Loading Success**:
|
||||
```
|
||||
========================================
|
||||
Progressive Loading Verification
|
||||
========================================
|
||||
|
||||
Step 1: Initial state
|
||||
On-demand mode: True
|
||||
Total available: 1000
|
||||
Loaded wildcards: 0
|
||||
|
||||
Step 2: Access __samples/flower__
|
||||
Result: rose
|
||||
Loaded wildcards: 1
|
||||
✓ PASS
|
||||
|
||||
Step 3: Access __dragon__
|
||||
Result: ancient dragon
|
||||
Loaded wildcards: 3
|
||||
✓ PASS
|
||||
|
||||
🎉 ALL TESTS PASSED
|
||||
Progressive on-demand loading verified!
|
||||
```
|
||||
|
||||
**Lazy Loading Success**:
|
||||
```
|
||||
========================================
|
||||
COMPARISON RESULTS
|
||||
========================================
|
||||
|
||||
Test: Wildcard List (before)
|
||||
✓ Results MATCH
|
||||
|
||||
Test: Simple Wildcard
|
||||
✓ Results MATCH
|
||||
|
||||
Test: Depth 3 Transitive
|
||||
✓ Results MATCH
|
||||
|
||||
🎉 ALL TESTS PASSED
|
||||
On-demand produces IDENTICAL results!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. Server Fails to Start
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
✗ Server failed to start
|
||||
curl: (7) Failed to connect
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# Check if port in use
|
||||
lsof -i :8188
|
||||
netstat -tlnp | grep 8188
|
||||
|
||||
# Kill existing processes
|
||||
pkill -f "python.*main.py"
|
||||
|
||||
# Increase startup wait time
|
||||
# In test script: sleep 15 → sleep 30
|
||||
```
|
||||
|
||||
#### 2. Module Not Found (Python)
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
ModuleNotFoundError: No module named 'modules'
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# Option 1: Run from ComfyUI directory
|
||||
cd /path/to/ComfyUI
|
||||
python3 custom_nodes/comfyui-impact-pack/tests/test_progressive_loading.py
|
||||
|
||||
# Option 2: Add to PYTHONPATH
|
||||
export PYTHONPATH=/path/to/ComfyUI/custom_nodes/comfyui-impact-pack:$PYTHONPATH
|
||||
python3 test_progressive_loading.py
|
||||
```
|
||||
|
||||
#### 3. On-Demand Mode Not Activating
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Using full cache mode.
|
||||
```
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
# View total size
|
||||
grep "Wildcard total size" /tmp/progressive_test.log
|
||||
|
||||
# Check cache limit
|
||||
grep "cache_limit_mb" impact-pack.ini
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# Force on-demand mode
|
||||
cat > impact-pack.ini << EOF
|
||||
[default]
|
||||
wildcard_cache_limit_mb = 0.5
|
||||
EOF
|
||||
```
|
||||
|
||||
#### 4. Tests Timeout
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Waiting for server startup...
|
||||
✗ Server failed to start
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# Check system resources
|
||||
free -h
|
||||
df -h
|
||||
|
||||
# View server logs
|
||||
tail -100 /tmp/progressive_test.log
|
||||
|
||||
# Manually test server
|
||||
cd /path/to/ComfyUI
|
||||
bash run.sh --port 8195
|
||||
|
||||
# Increase timeout in test
|
||||
# sleep 15 → sleep 60
|
||||
```
|
||||
|
||||
#### 5. Results Don't Match
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
✗ Results DIFFER
|
||||
```
|
||||
|
||||
**Debug**:
|
||||
```bash
|
||||
# Compare results
|
||||
diff /tmp/result_full_cache_simple.json /tmp/result_on_demand_simple.json
|
||||
|
||||
# Check seeds are same
|
||||
grep "seed" /tmp/result_*.json
|
||||
|
||||
# Verify same wildcard files used
|
||||
ls -la wildcards/samples/flower.txt
|
||||
```
|
||||
|
||||
**File Bug Report**:
|
||||
- Wildcard text
|
||||
- Seed value
|
||||
- Full cache result
|
||||
- On-demand result
|
||||
- Server logs
|
||||
|
||||
#### 6. Slow Performance
|
||||
|
||||
**Symptoms**:
|
||||
- Tests take much longer than expected
|
||||
- Server startup > 2 minutes
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
# Wildcard size
|
||||
du -sh wildcards/
|
||||
|
||||
# Disk I/O
|
||||
iostat -x 1 5
|
||||
|
||||
# System resources
|
||||
top
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
- Use SSD (not HDD)
|
||||
- Reduce wildcard size
|
||||
- Increase cache limit (use full cache mode)
|
||||
- Close other applications
|
||||
|
||||
---
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
### Expected Performance
|
||||
|
||||
**Small Dataset (< 50MB)**:
|
||||
```
|
||||
Mode: Full cache
|
||||
Startup: < 10 seconds
|
||||
Memory: ~50MB
|
||||
First access: Instant
|
||||
```
|
||||
|
||||
**Medium Dataset (50MB - 1GB)**:
|
||||
```
|
||||
Mode: On-demand
|
||||
Startup: < 30 seconds
|
||||
Memory: < 200MB initial
|
||||
First access: 10-50ms per wildcard
|
||||
```
|
||||
|
||||
**Large Dataset (10GB+)**:
|
||||
```
|
||||
Mode: On-demand
|
||||
Startup: < 1 minute
|
||||
Memory: < 100MB initial
|
||||
First access: 10-50ms per wildcard
|
||||
Memory growth: Progressive
|
||||
```
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
**For Faster Tests**:
|
||||
1. Use smaller wildcard dataset
|
||||
2. Run specific tests (not all)
|
||||
3. Use manual server (keep running)
|
||||
4. Skip sleep times (if server already running)
|
||||
|
||||
**For Large Datasets**:
|
||||
1. Verify on-demand mode activates
|
||||
2. Monitor `/list/loaded` to track memory
|
||||
3. Use SSD for file storage
|
||||
4. Organize wildcards into subdirectories
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
### Adding New Tests
|
||||
|
||||
**1. Create Test File**:
|
||||
```bash
|
||||
touch tests/test_new_feature.sh
|
||||
chmod +x tests/test_new_feature.sh
|
||||
```
|
||||
|
||||
**2. Test Template**:
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Test: New Feature
|
||||
# Purpose: Verify new feature works correctly
|
||||
|
||||
set -e
|
||||
|
||||
PORT=8XXX
|
||||
IMPACT_DIR="/path/to/comfyui-impact-pack"
|
||||
|
||||
# Setup config
|
||||
cat > impact-pack.ini << EOF
|
||||
[default]
|
||||
wildcard_cache_limit_mb = 50
|
||||
EOF
|
||||
|
||||
# Start server
|
||||
cd /path/to/ComfyUI
|
||||
bash run.sh --port $PORT > /tmp/test_new.log 2>&1 &
|
||||
sleep 15
|
||||
|
||||
# Test
|
||||
RESULT=$(curl -s http://127.0.0.1:$PORT/impact/wildcards/list)
|
||||
|
||||
# Validate
|
||||
if [ "$RESULT" = "expected" ]; then
|
||||
echo "✅ PASS"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ FAIL"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
**3. Update Documentation**:
|
||||
- Add test description to this README
|
||||
- Update test count
|
||||
- Add to appropriate category
|
||||
|
||||
### Testing Guidelines
|
||||
|
||||
**Test Structure**:
|
||||
1. Clear purpose statement
|
||||
2. Setup (config, wildcards)
|
||||
3. Execution (API calls, processing)
|
||||
4. Validation (assertions, comparisons)
|
||||
5. Cleanup (kill servers, restore config)
|
||||
|
||||
**Good Practices**:
|
||||
- Use unique port numbers
|
||||
- Clean up background processes
|
||||
- Provide clear success/failure messages
|
||||
- Log to `/tmp/` for debugging
|
||||
- Use deterministic seeds
|
||||
- Test both modes (full cache + on-demand)
|
||||
|
||||
---
|
||||
|
||||
## Reference
|
||||
|
||||
### Test Files Quick Reference
|
||||
|
||||
```bash
|
||||
# Progressive loading
|
||||
test_progressive_ondemand.sh # Integration test
|
||||
test_progressive_loading.py # Unit test
|
||||
|
||||
# Lazy loading
|
||||
test_lazy_load_api.sh # Integration test
|
||||
test_wildcard_lazy_loading.py # Unit test
|
||||
|
||||
# Sequential/transitive
|
||||
test_sequential_loading.sh # Integration test
|
||||
find_transitive_wildcards.sh # Utility
|
||||
|
||||
# Features
|
||||
test_versatile_prompts.sh # Comprehensive features
|
||||
test_wildcard_features.sh # Core features
|
||||
test_wildcard_consistency.sh # Consistency
|
||||
|
||||
# Validation
|
||||
test_wildcard_final.py # Final validation
|
||||
test_lazy_load_verification.py # Lazy load verification
|
||||
```
|
||||
|
||||
### Documentation
|
||||
|
||||
- **System Overview**: `../docs/WILDCARD_SYSTEM_OVERVIEW.md`
|
||||
- **Testing Guide**: `../docs/WILDCARD_TESTING_GUIDE.md`
|
||||
|
||||
### API Endpoints
|
||||
|
||||
```
|
||||
GET /impact/wildcards/list # All available wildcards
|
||||
GET /impact/wildcards/list/loaded # Actually loaded (progressive)
|
||||
POST /impact/wildcards # Process wildcard text
|
||||
GET /impact/wildcards/refresh # Reload all wildcards
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2024-11-17
|
||||
**Total Tests**: 11 files, 100+ scenarios
|
||||
**Coverage**: ~95% of wildcard features
|
||||
Reference in New Issue
Block a user