Flux Depth Control Header Image

This image is generated by AI

Besides using edges as a control reference, there is another way to control called depth map. This control method can provide some positional depth information to the model, so as to generate an image with the same content position depth as the original image. For example, as shown in the figure below, I input an image to the Flux model, and then let the Flux model generate an image with the same content position as the original image. You can see that the generated image, except for a slight change in expression (mainly because my Prompt is not written well), the overall position relationship of the characters and the posture of the characters are almost the same as the original image.
Flux Depth Control

​
1. Download Flux Depth Model

Similar to Canny, there are currently two types of Flux Depth models, one is the Flux Depth LoRA implemented by Flux official, and the other is the Flux Depth Controlnet implemented by other open-source organizations. The former is based on the Control LoRA technology, while the latter is based on the ControlNet technology. You can all download them and try them out:
  • Flux official provides two Depth LoRA model files, you can choose according to your needs:
    • One is the model that includes Flux Depth LoRA and Flux Dev. You can download this model from here, and then place it in the /models/diffusion_models/ folder. However, because this model includes the Flux Dev model, it is quite large, about 24GB. But the good thing is that you only need to load one model when loading the model.
    • If you don’t want to download such a large model, you can download the separate Flux Depth LoRA model from here, and then place it in the models/loras/ folder. This model file is only about 1.24GB.
  • There are currently two open-source organizations that have implemented the Flux Depth Controlnet model, you can choose according to your needs:
    • The first is the Depth ControlNet model jointly developed by Shakker-Labs and InstantX. You need to download this model and place it in the models/controlnet/ folder. Also, note that this file can be renamed when downloading, renamed as instantx_flux_depth.safetensors, to facilitate subsequent use.
    • The other is the Flux Depth controlnet v2 model developed by XLabs. You also need to download this model and place it in the models/controlnet/ folder.

​
2. Flux Depth LoRA Workflow

If you download the version that combines the LoRA and Dev model, then you only need to modify the workflow based on the Flux Edge Control in the previous chapter. Or go to Comflowy to download this workflow and import it for use. The method is very simple, you just need to replace the flux Canny in the ① Diffusion node group with the Depth model, and then replace the Canny node with any Depth node (you can use any Depth node you are familiar with) (Figure ②). Finally, connect the Depth node and the Load Image node. If you want to preview the effect of the Depth node, you can connect the Depth node and the Preview Image node, so you can see the depth map output by the Depth node.
Flux Depth Control
If you download the separate Flux Depth LoRA model, then you just need to add a LoraLoaderModelOnly node (Figure ②) after the Load Diffusion Model node (Figure ①). Then replace the flux dev depth model in the Diffusion node group with flux dev, and then select the Flux Depth LoRA model you downloaded in the LoraLoaderModelOnly node. The effect of the two will basically not be too different.
Flux Depth Control

​
3. Flux Depth Controlnet Workflow

​
3.1 InstantX Version

First, because running the ControlNet model will occupy extra GPU memory, I recommend you load the GGUF version Flux workflow in the Flux ComfyUI workflow, and then modify it based on this workflow. Or go to Comflowy to download this workflow and import it for use. This version requires:
  1. Add the Apply ControlNet with VAE node (as shown in Figure ①). Then connect it with Load VAE (Figure ②), Load ControlNet Model (Figure ②) and any Depth node (Figure ③).
  2. Select the downloaded Flux Depth Controlnet model in the Load ControlNet Model node.
  3. To ensure that the generated image is the same size as the input Depth map, I also added a Get Image Size node (Figure ④), and then connected it with the SD3LatentImage node (Figure ⑤). So the generated image size is the same as the input Depth map size.
Flux Depth Control
Also, if you don’t want to use the GGUF version, you can also use the FP8 version provided by ComfyUI, just need to remove the Load VAE and DualCLIPLoader nodes, and replace them with the Load Checkpoint node. Then select the FP8 version of the Flux model.
Flux Depth Control
Personally, I prefer the Flux official Depth LoRA model, its effect is better.

​
3.2 XLabs Version

Using the XLabs version of the Flux Depth Controlnet model requires using the XLabs developed plugin. This is their Github plugin address. You can install this plugin through ComfyUI’s ComfyUI-Manager. For detailed installation methods, please refer to Install ComfyUI Plugin article. After installation, you can modify the workflow of the Shakker-Labs version. Or go to Comflowy to download this workflow and import it for use. The modification is not too difficult:
FigureDescription
①Replace the KSampler node with the Xlabs Sampler node. You can see that this node has an additional controlnet_condition input.
②I replaced the CLIPTextEncode node with the CLIPTextEncodeFlux node. You don’t need to modify it. I just want to tell you that there are other Flux CLIP nodes. This node can control Clip l and t5xxl separately.
③Replace the Apply ControlNet VAE node with the Apply Flux ControlNet node.
④Replace the Load ControlNet Model node with the Load Flux ControlNet node. And select the downloaded Flux Depth Controlnet model.
⑤I replaced a Depth node. You can use any Depth node you are familiar with.
⑥SD3LatentImage node I also used the Empty Latent Image node. This is also to show that this node can be used with the most basic version besides the SD3 version.
Flux Depth Control

​
4. Flux Depth API Workflow

If your computer performance is not sufficient to run the Flux Depth model, you can also try using Comflowy’s Depth API node in ComfyUI. Of course, you can also directly use the Flux Depth API node in Comflowy, with a simple connection method that requires just one node, and it also supports Flux Pro and Dev versions.
Note that since this node uses an API, you will need to pay the corresponding fees when using it.
Flux Depth Control