ControlNet Usage Guide

ControlNet gives you precise control over image generation by conditioning on structural inputs like edges, depth maps, poses, and more. Instead of relying solely on text prompts, you guide the generation with visual references.

What is ControlNet?

ControlNet is an extension to Stable Diffusion that accepts control images:

Canny edges: Line art and contours
Depth maps: 3D structure information
Pose: Human skeleton keypoints (OpenPose)
Normal maps: Surface orientation
Scribbles: Rough sketches
Segmentation: Semantic regions

The model generates images that match both your text prompt and the structure from the control image.

Prerequisites

ControlNet model files in models/controlnet/
Preprocessor models (optional, for automatic control image generation)
Input image or pre-processed control map

Download ControlNet models from Hugging Face (lllyasviel/ControlNet).

Basic Workflow

Load your checkpoint

Add CheckpointLoaderSimple to load your base Stable Diffusion model. ControlNet works with SD 1.5, SD 2.1, and SDXL variants.

Load ControlNet

Add ControlNetLoader node

Select your ControlNet model:

control_v11p_sd15_canny.pth (edge detection)
control_v11f1p_sd15_depth.pth (depth)
control_v11p_sd15_openpose.pth (pose)
Many others available

Prepare control image

Option A - Use existing control image:

Add LoadImage node

Load a pre-processed control image (edges, depth map, etc.)

Option B - Auto-generate from image:

Add LoadImage for source image

Add preprocessor node (Canny, Depth, etc.)

Connect image to preprocessor

Note: Preprocessor nodes vary by custom node packs. Many use nodes like “CannyEdgePreprocessor” or built-in image processing.

Apply ControlNet

Add ControlNetApplyAdvanced node

Make connections:

positive: from positive CLIPTextEncode
negative: from negative CLIPTextEncode
control_net: from ControlNetLoader
image: from LoadImage or preprocessor

Configure parameters:

strength: 0.5-1.5 (how strongly to follow control image)
start_percent: 0.0 (when to start applying control)
end_percent: 1.0 (when to stop applying control)

Generate as normal

Use outputs from ControlNetApplyAdvanced as conditioning for KSampler

Generate with EmptyLatentImage (or VAEEncode for img2img)

Decode and save

Complete Workflow JSON

{
  "checkpoint": {
    "class_type": "CheckpointLoaderSimple",
    "inputs": {
      "ckpt_name": "v1-5-pruned-emaonly.safetensors"
    }
  },
  "controlnet_loader": {
    "class_type": "ControlNetLoader",
    "inputs": {
      "control_net_name": "control_v11p_sd15_canny.pth"
    }
  },
  "load_control_image": {
    "class_type": "LoadImage",
    "inputs": {
      "image": "canny_edges.png"
    }
  },
  "positive": {
    "class_type": "CLIPTextEncode",
    "inputs": {
      "clip": ["checkpoint", 1],
      "text": "beautiful anime character, colorful, high quality, detailed"
    }
  },
  "negative": {
    "class_type": "CLIPTextEncode",
    "inputs": {
      "clip": ["checkpoint", 1],
      "text": "blurry, low quality, distorted"
    }
  },
  "controlnet_apply": {
    "class_type": "ControlNetApplyAdvanced",
    "inputs": {
      "positive": ["positive", 0],
      "negative": ["negative", 0],
      "control_net": ["controlnet_loader", 0],
      "image": ["load_control_image", 0],
      "strength": 1.0,
      "start_percent": 0.0,
      "end_percent": 1.0
    }
  },
  "empty_latent": {
    "class_type": "EmptyLatentImage",
    "inputs": {
      "width": 512,
      "height": 768,
      "batch_size": 1
    }
  },
  "sampler": {
    "class_type": "KSampler",
    "inputs": {
      "model": ["checkpoint", 0],
      "positive": ["controlnet_apply", 0],
      "negative": ["controlnet_apply", 1],
      "latent_image": ["empty_latent", 0],
      "seed": 42,
      "steps": 25,
      "cfg": 8.0,
      "sampler_name": "euler",
      "scheduler": "normal",
      "denoise": 1.0
    }
  },
  "vae_decode": {
    "class_type": "VAEDecode",
    "inputs": {
      "samples": ["sampler", 0],
      "vae": ["checkpoint", 2]
    }
  },
  "save": {
    "class_type": "SaveImage",
    "inputs": {
      "images": ["vae_decode", 0],
      "filename_prefix": "controlnet"
    }
  }
}

ControlNet Types

Canny Edge Detection

Best for: Line art, architectural drawings, precise outlines Control image: Black background with white edges Use cases:

Converting sketches to full illustrations
Maintaining architectural structure
Preserving composition while changing style

Strength: 0.8-1.2

Depth Map

Best for: 3D structure, perspective control Control image: Grayscale depth map (closer = brighter) Use cases:

Maintaining scene depth
Generating similar compositions
3D-to-2D conversion

Strength: 0.6-1.0

OpenPose

Best for: Human pose control Control image: Stick figure skeleton Use cases:

Character pose consistency
Animation frame generation
Pose transfer between characters

Strength: 0.8-1.2

Scribble

Best for: Rough sketches, quick ideas Control image: Simple line drawings Use cases:

Concept exploration
Quick ideation
Loose composition control

Strength: 0.5-0.9

Segmentation

Best for: Color/region blocking Control image: Colored regions for different semantic areas Use cases:

Precise region control
Complex compositions
Scene layout planning

Strength: 0.7-1.0

Normal Map

Best for: Surface detail, lighting consistency Control image: RGB normal map Use cases:

Detailed surface structure
Maintaining 3D form
Lighting-consistent generation

Strength: 0.5-0.8

Parameter Guide

Strength

Controls how strongly the ControlNet influences generation:

0.3-0.5: Subtle guidance, more creative freedom
0.6-0.8: Moderate control, balanced
0.9-1.2: Strong adherence to control image
1.3-1.5: Very strict, may reduce quality
> 1.5: Can cause artifacts

Start/End Percent

Control when ControlNet is active during denoising: Full control (default):

{
  "start_percent": 0.0,
  "end_percent": 1.0
}

Structure first, details later:

{
  "start_percent": 0.0,
  "end_percent": 0.7
}

Applies control for composition, lets model add creative details. Refinement only:

{
  "start_percent": 0.3,
  "end_percent": 1.0
}

Lets model establish basic structure, then applies control.

Multiple ControlNets

Combine multiple ControlNets for complex control:

Load multiple ControlNetLoader nodes
Chain ControlNetApplyAdvanced nodes:
- First: Apply ControlNet A to base conditioning
- Second: Apply ControlNet B to output of first
Use final output for KSampler

Example: Depth + Pose

{
  "controlnet_depth_apply": {
    "class_type": "ControlNetApplyAdvanced",
    "inputs": {
      "positive": ["positive", 0],
      "negative": ["negative", 0],
      "control_net": ["depth_loader", 0],
      "image": ["depth_image", 0],
      "strength": 0.8,
      "start_percent": 0.0,
      "end_percent": 1.0
    }
  },
  "controlnet_pose_apply": {
    "class_type": "ControlNetApplyAdvanced",
    "inputs": {
      "positive": ["controlnet_depth_apply", 0],
      "negative": ["controlnet_depth_apply", 1],
      "control_net": ["pose_loader", 0],
      "image": ["pose_image", 0],
      "strength": 1.0,
      "start_percent": 0.0,
      "end_percent": 0.8
    }
  }
}

ControlNet with Img2Img

Combine ControlNet with image-to-image:

Use VAEEncode instead of EmptyLatentImage
Set denoise 0.6-0.9
Control image guides structure
Source image provides style/details

This is powerful for style transfer with structural consistency.

ControlNet with Inpainting

Apply ControlNet to inpainting:

Use VAEEncodeForInpaint or InpaintModelConditioning
Apply ControlNet to conditioning
Control image guides the inpainted region

Use case: Replace an object while maintaining scene perspective (depth ControlNet).

Advanced Techniques

Regional ControlNet

Use ConditioningSetMask to apply ControlNet only to specific regions:

{
  "regional_control": {
    "class_type": "ConditioningSetMask",
    "inputs": {
      "conditioning": ["controlnet_apply", 0],
      "mask": ["region_mask", 0],
      "strength": 1.0,
      "set_cond_area": "mask bounds"
    }
  }
}

Strength Scheduling

Vary strength during generation using start/end_percent: Strong structure, creative details:

strength: 1.2
start_percent: 0.0
end_percent: 0.5

Soft guidance throughout:

strength: 0.6
start_percent: 0.0
end_percent: 1.0

Control Image Preprocessing

For best results:

Match control image resolution to generation size
Use high-contrast edges for Canny
Clean poses for OpenPose
Smooth depth maps for better results

Common Use Cases

Architecture from Floorplan

Create line drawing of floor plan
Use Canny ControlNet, strength 1.0-1.2
Prompt: architectural style, materials, lighting
Result: Photorealistic render following layout

Character Pose Consistency

Extract pose from reference with OpenPose
Use pose ControlNet, strength 0.9-1.1
Generate multiple characters in same pose
Perfect for animation frames or style variations

Photo to Illustration

Source photo → Canny edges
Canny ControlNet, strength 0.7-0.9
Prompt artistic style
Result: Illustration with original composition

3D Scene Rendering

Export depth map from 3D software
Depth ControlNet, strength 0.8
Prompt environment details
Result: Photorealistic render with 3D structure

Troubleshooting

Too rigid/stiff results

Lower strength (try 0.6-0.8)
Reduce end_percent (try 0.7)
Use less precise control type (scribble vs canny)

Control not being followed

Increase strength (try 1.0-1.2)
Check control image quality
Ensure control image matches generation size
Increase CFG scale

Artifacts or distortion

Lower strength (try 0.7-0.9)
Increase steps (try 30-40)
Check control image for noise/errors
Try different sampler

Wrong ControlNet type

Canny edges appear blurry → regenerate with higher threshold
Depth looks flat → use better depth estimation
Pose ignored → ensure skeleton is clear and complete

Best Practices

Match resolutions: Control image should match target size
Start conservative: Begin with strength 0.8, adjust up/down
Clean inputs: High-quality control images = better results
Combine wisely: 2-3 ControlNets max, more can conflict
Prompt alignment: Text should complement control image
Test settings: Different models respond differently to strength
Use end_percent: Often better to release control for final details

Model Compatibility

SD 1.5: Most ControlNet models available
SD 2.1: Fewer models, similar usage
SDXL: SDXL-specific ControlNets needed
SD3: Limited ControlNet support currently

Always match ControlNet version to base model version.

Next Steps

Experiment with different ControlNet types
Combine multiple ControlNets for complex scenarios
Try ControlNet with LoRAs for style + structure control
Explore preprocessors for automatic control image generation
Create custom control images for precise artistic control

Tutorials

Integration

Troubleshooting

ControlNet Usage Guide

What is ControlNet?

Prerequisites

Basic Workflow

Complete Workflow JSON

ControlNet Types

Canny Edge Detection

Depth Map

OpenPose

Scribble

Segmentation

Normal Map

Parameter Guide

Strength

Start/End Percent

Multiple ControlNets

ControlNet with Img2Img

ControlNet with Inpainting

Advanced Techniques

Regional ControlNet

Strength Scheduling

Control Image Preprocessing

Common Use Cases

Architecture from Floorplan

Character Pose Consistency

Photo to Illustration

3D Scene Rendering

Troubleshooting

Too rigid/stiff results

Control not being followed

Artifacts or distortion

Wrong ControlNet type

Best Practices

Model Compatibility

Next Steps

Build docs developers (and LLMs) love

Tutorials

Integration

Troubleshooting

​What is ControlNet?

​Prerequisites

​Basic Workflow

​Complete Workflow JSON

​ControlNet Types

​Canny Edge Detection

​Depth Map

​OpenPose

​Scribble

​Segmentation

​Normal Map

​Parameter Guide

​Strength

​Start/End Percent

​Multiple ControlNets

​ControlNet with Img2Img

​ControlNet with Inpainting

​Advanced Techniques

​Regional ControlNet

​Strength Scheduling

​Control Image Preprocessing

​Common Use Cases

​Architecture from Floorplan

​Character Pose Consistency

​Photo to Illustration

​3D Scene Rendering

​Troubleshooting

​Too rigid/stiff results

​Control not being followed

​Artifacts or distortion

​Wrong ControlNet type

​Best Practices

​Model Compatibility

​Next Steps

Build docs developers (and LLMs) love

What is ControlNet?

Prerequisites

Basic Workflow

Complete Workflow JSON

ControlNet Types

Canny Edge Detection

Depth Map

OpenPose

Scribble

Segmentation

Normal Map

Parameter Guide

Strength

Start/End Percent

Multiple ControlNets

ControlNet with Img2Img

ControlNet with Inpainting

Advanced Techniques

Regional ControlNet

Strength Scheduling

Control Image Preprocessing

Common Use Cases

Architecture from Floorplan

Character Pose Consistency

Photo to Illustration

3D Scene Rendering

Troubleshooting

Too rigid/stiff results

Control not being followed

Artifacts or distortion

Wrong ControlNet type

Best Practices

Model Compatibility

Next Steps