lllyasviel

ControlNet is a neural network structure to control diffusion models by adding extra conditions. This checkpoint corresponds to the ControlNet conditioned on Canny edges. It can be used in combination with Stable Diffusion. Model Details - Developed by: Lvmin Zhang, Maneesh Agrawala - Model type: Diffusion-based text-to-image generation model - Language(s): English - License: The CreativeML OpenRAIL M license is an Open RAIL M license, adapted from the work that BigScience and the RAIL Initiative are jointly carrying in the area of responsible AI licensing. See also the article about the BLOOM Open RAIL license on which our license is based. - Resources for more information: GitHub Repository, Paper. - Cite as: @misc{zhang2023adding, title={Adding Conditional Control to Text-to-Image Diffusion Models}, author={Lvmin Zhang and Maneesh Agrawala}, year={2023}, eprint={2302.05543}, archivePrefix={arXiv}, primaryClass={cs.CV} } Controlnet was proposed in Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang, Maneesh Agrawala. We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small ( Trained with canny edge detection | A monochrome image with white edges on a black background.| | | |lllyasviel/sd-controlnet-depth Trained with Midas depth estimation |A grayscale image with black representing deep areas and white representing shallow areas.| | | |lllyasviel/sd-controlnet-hed Trained with HED edge detection (soft edge) |A monochrome image with white soft edges on a black background.| | | |lllyasviel/sd-controlnet-mlsd Trained with M-LSD line detection |A monochrome image composed only of white straight lines on a black background.| | | |lllyasviel/sd-controlnet-normal Trained with normal map |A normal mapped image.| | | |lllyasviel/sd-controlnetopenpose Trained with OpenPose bone image |A OpenPose bone image.| | | |lllyasviel/sd-controlnetscribble Trained with human scribbles |A hand-drawn monochrome image with white outlines on a black background.| | | |lllyasviel/sd-controlnetseg Trained with semantic segmentation |An ADE20K's segmentation protocol image.| | | It is recommended to use the checkpoint with Stable Diffusion v1-5 as the checkpoint has been trained on it. Experimentally, the checkpoint can be used with other diffusion models such as dreamboothed stable diffusion. Note: If you want to process an image to create the auxiliary conditioning, external dependencies are required as shown below: The canny edge model was trained on 3M edge-image, caption pairs. The model was trained for 600 GPU-hours with Nvidia A100 80G using Stable Diffusion 1.5 as a base model. For more information, please also have a look at the official ControlNet Blog Post.

This is the forward-only FramePack-F1 with `f1k1xtdf16k4f2k2f1k1g9`, trained with anti-drifting regulations.

This is a GGUF mirror of black-forest-labs/FLUX.1-schnell quantized version.

This is the pretrained weights and some other detector weights of ControlNet. - The ControlNet+SD1.5 model to control SD using canny edge detection. - The ControlNet+SD1.5 model to control SD using Midas depth estimation. - The ControlNet+SD1.5 model to control SD using HED edge detection (soft edge). - The ControlNet+SD1.5 model to control SD using M-LSD line detection (will also work with traditional Hough transform). - The ControlNet+SD1.5 model to control SD using normal map. Best to use the normal map generated by that Gradio app. Other normal maps may also work as long as the direction is correct (left looks red, right looks blue, up looks green, down looks purple). - The ControlNet+SD1.5 model to control SD using OpenPose pose detection. Directly manipulating pose skeleton should also work. - The ControlNet+SD1.5 model to control SD using human scribbles. The model is trained with boundary edges with very strong data augmentation to simulate boundary lines similar to that drawn by human. - The ControlNet+SD1.5 model to control SD using semantic segmentation. The protocol is ADE20k. - Third-party model: Openpose’s pose detection model. - Third-party model: Openpose’s hand detection model. ControlNet/annotator/ckpts/dpthybrid-midas-501f0c75.pt - Third-party model: M-LSD’s another smaller detection model (we do not use this one). - Third-party model: Uniformer semantic segmentation. Special Thank to the great project - Mikubill' A1111 Webui Plugin ! We also thank Hysts for making Gradio demo in Hugging Face Space as well as more than 65 models in that amazing Colab list! Thank haofanwang for making ControlNet-for-Diffusers! We also thank all authors for making Controlnet DEMOs, including but not limited to fffiloni, other-model, ThereforeGames, RamAnanth1, etc! The model should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.

Collection of community SD control models for users to download flexibly. All files are already float16 and in safetensor format. files = { 'diffusersxlcannysmall.safetensors': 'https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0-small/resolve/main/diffusionpytorchmodel.bin', 'diffusersxlcannymid.safetensors': 'https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0-mid/resolve/main/diffusionpytorchmodel.bin', 'diffusersxlcannyfull.safetensors': 'https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0/resolve/main/diffusionpytorchmodel.bin', 'diffusersxldepthsmall.safetensors': 'https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0-small/resolve/main/diffusionpytorchmodel.bin', 'diffusersxldepthmid.safetensors': 'https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0-mid/resolve/main/diffusionpytorchmodel.bin', 'diffusersxldepthfull.safetensors': 'https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0/resolve/main/diffusionpytorchmodel.bin', 'thibaudxlopenpose.safetensors': 'https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0/resolve/main/OpenPoseXL2.safetensors', 'thibaudxlopenpose256lora.safetensors': 'https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0/resolve/main/control-lora-openposeXL2-rank256.safetensors', 'sargeztxldepthfaidvidit.safetensors': 'https://huggingface.co/SargeZT/controlnet-sd-xl-1.0-depth-faid-vidit/resolve/main/diffusionpytorchmodel.bin', 'sargeztxldepthzeed.safetensors': 'https://huggingface.co/SargeZT/controlnet-sd-xl-1.0-depth-zeed/resolve/main/diffusionpytorchmodel.bin', 'sargeztxldepth.safetensors': 'https://huggingface.co/SargeZT/controlnet-v1e-sdxl-depth/resolve/main/diffusionpytorchmodel.bin', 'sargeztxlsoftedge.safetensors': 'https://huggingface.co/SargeZT/controlnet-sd-xl-1.0-softedge-dexined/resolve/main/controlnet-sd-xl-1.0-softedge-dexined.safetensors', 'saixlcanny128lora.safetensors': 'https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank128/control-lora-canny-rank128.safetensors', 'saixlcanny256lora.safetensors': 'https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank256/control-lora-canny-rank256.safetensors', 'saixldepth128lora.safetensors': 'https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank128/control-lora-depth-rank128.safetensors', 'saixldepth256lora.safetensors': 'https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank256/control-lora-depth-rank256.safetensors', 'saixlsketch128lora.safetensors': 'https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank128/control-lora-sketch-rank128-metadata.safetensors', 'saixlsketch256lora.safetensors': 'https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank256/control-lora-sketch-rank256.safetensors', 'saixlrecolor128lora.safetensors': 'https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank128/control-lora-recolor-rank128.safetensors', 'saixlrecolor256lora.safetensors': 'https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank256/control-lora-recolor-rank256.safetensors', 'ioclabsd15recolor.safetensors': 'https://huggingface.co/ioclab/controlv1psd15brightness/resolve/main/diffusionpytorchmodel.safetensors', 't2i-adapterxlcanny.safetensors': 'https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/modelsXL/adapter-xl-canny.pth', 't2i-adapterxlopenpose.safetensors': 'https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/modelsXL/adapter-xl-openpose.pth', 't2i-adapterxlsketch.safetensors': 'https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/modelsXL/adapter-xl-sketch.pth', 'ip-adaptersd15plus.safetensors': 'https://huggingface.co/h94/IP-Adapter/resolve/main/models/ip-adapter-plussd15.bin', 'ip-adaptersd15.safetensors': 'https://huggingface.co/h94/IP-Adapter/resolve/main/models/ip-adaptersd15.bin', 'ip-adapterxl.safetensors': 'https://huggingface.co/h94/IP-Adapter/resolve/main/sdxlmodels/ip-adaptersdxl.bin', 'kohyacontrollllitexldepthanime.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01008016esdxldepthanime.safetensors', 'kohyacontrollllitexlcannyanime.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01032064esdxlcannyanime.safetensors', 'kohyacontrollllitexlscribbleanime.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01032064esdxlfakescribbleanime.safetensors', 'kohyacontrollllitexlopenposeanime.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01032064esdxlposeanime.safetensors', 'kohyacontrollllitexlopenposeanimev2.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01032064esdxlposeanimev2500-1000.safetensors', 'kohyacontrollllitexlbluranimebeta.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01016032esdxlbluranimebeta.safetensors', 'kohyacontrollllitexlblur.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01032064esdxlblur-500-1000.safetensors', 'kohyacontrollllitexlbluranime.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01032064esdxlblur-anime500-1000.safetensors', 'kohyacontrollllitexlcanny.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01032064esdxlcanny.safetensors', 'kohyacontrollllitexldepth.safetensors': 'https://huggingface.co/kohya-ss/controlnet-lllite/resolve/main/controllllitev01032064esdxldepth500-1000.safetensors', 't2i-adapterdiffusersxlcanny.safetensors': 'https://huggingface.co/TencentARC/t2i-adapter-canny-sdxl-1.0/resolve/main/diffusionpytorchmodel.safetensors', 't2i-adapterdiffusersxllineart.safetensors': 'https://huggingface.co/TencentARC/t2i-adapter-lineart-sdxl-1.0/resolve/main/diffusionpytorchmodel.safetensors', 't2i-adapterdiffusersxldepthmidas.safetensors': 'https://huggingface.co/TencentARC/t2i-adapter-depth-midas-sdxl-1.0/resolve/main/diffusionpytorchmodel.safetensors', 't2i-adapterdiffusersxlopenpose.safetensors': 'https://huggingface.co/TencentARC/t2i-adapter-openpose-sdxl-1.0/resolve/main/diffusionpytorchmodel.safetensors', 't2i-adapterdiffusersxldepthzoe.safetensors': 'https://huggingface.co/TencentARC/t2i-adapter-depth-zoe-sdxl-1.0/resolve/main/diffusionpytorchmodel.safetensors', 't2i-adapterdiffusersxlsketch.safetensors': 'https://huggingface.co/TencentARC/t2i-adapter-sketch-sdxl-1.0/resolve/main/diffusionpytorchmodel.safetensors', If you download the files from raw URL, you may need to rename them. However, files in https://huggingface.co/lllyasviel/sdcontrolcollection/tree/main are already renamed and can be directly downloaded. Feel free to contact us if you are author of any listed models and you want some models to be removed/added (by opening an issue in this HuggingFace page).

Main page: https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/981 V2 is quantized in a better way to turn off the second stage of double quant. V2 is 0.5 GB larger than the previous version, since the chunk 64 norm is now stored in full precision float32, making it much more precise than the previous version. Also, since V2 does not have second compression stage, it now has less computation overhead for on-the-fly decompression, making the inference a bit faster. Main model in bnb-nf4 (v1 with chunk 64 norm in nf4, v2 with chunk 64 norm in float32)