Problem with controlnet(sd-controlnet-canny)

[Dmitr15]

Hello. I have another problem, now it's related to controlnet.

I copied the code from this tutorial https://pjoshi15.com/how-to-perform-style-transfer-with-stable-diffusion-and-python/

I have next problems:

Traceback (most recent call last):
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\utils_http.py", line 406, in hf_raise_for_status
response.raise_for_status()
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\requests\models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/None/resolve/main/config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\configuration_utils.py", line 379, in load_config
config_file = hf_hub_download(
^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\utils_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\file_download.py", line 860, in hf_hub_download
return _hf_hub_download_to_cache_dir(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\file_download.py", line 967, in _hf_hub_download_to_cache_dir
_raise_on_head_call_error(head_call_error, force_download, local_files_only)
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\file_download.py", line 1482, in _raise_on_head_call_error
raise head_call_error
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\file_download.py", line 1374, in _get_metadata_or_catch_error
metadata = get_hf_file_metadata(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\utils_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\file_download.py", line 1294, in get_hf_file_metadata
r = _request_wrapper(
^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\file_download.py", line 278, in _request_wrapper
response = _request_wrapper(
^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\file_download.py", line 302, in _request_wrapper
hf_raise_for_status(response)
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\utils_http.py", line 454, in hf_raise_for_status
raise _format(RepositoryNotFoundError, message, response) from e
huggingface_hub.errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-67b37cb3-37fe29d90fe559b83ae5f1b7;2c044edd-b4d0-4bc8-8164-c8714e6ddbed)

Repository Not Found for url: https://huggingface.co/None/resolve/main/config.json.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\style_change.py", line 15, in
controlnet = ControlNetModel.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\utils_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\models\modeling_utils.py", line 687, in from_pretrained
config, unused_kwargs, commit_hash = cls.load_config(
^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\huggingface_hub\utils_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\configuration_utils.py", line 394, in load_config
raise EnvironmentError(
OSError: None is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with token or log in with huggingface-cli login.

My code:

import torch
from controlnet_aux import CannyDetector
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from diffusers.utils import load_image, make_image_grid
import os

MMIX_MODEL_PATH = os.getenv('MMIX_MODEL_PATH')
IP_ADAPTER_PATH = os.getenv('IP_ADAPTER')
CANNY_MODEL_PATH=os.getenv('CANNY_MODEL')

img1 = 'C:\Users\dovsy\Downloads\image-4.png'
img2 = 'C:\Users\dovsy\Downloads\image-3.png'

controlnet = ControlNetModel.from_pretrained(
CANNY_MODEL_PATH,
torch_dtype=torch.float16
#varient="fp16"
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
MMIX_MODEL_PATH,
controlnet=controlnet,
torch_dtype=torch.float16)

pipe.load_ip_adapter(
IP_ADAPTER_PATH+'//models//ip-adapter_sd15.bin',
#subfolder="//models",
#weight_name="ip-adapter_sd15.bin"
)

ip_adap_img = load_image(img2)
img = load_image(img1).resize((768, 768))

canny = CannyDetector()
canny_img = canny(img, detect_resolution=512, image_resolution=768)

prompt = """(photorealistic:1.2), raw, masterpiece, high quality, 8k, girl wearing hoodie, headphones, hands in pocket"""
negative_prompt = "low quality, ugly, mutated"

pipe.set_ip_adapter_scale(0.5)

images = pipe(prompt = prompt,
negative_prompt = negative_prompt,
height = 768,
width = 768,
ip_adapter_image = ip_adap_img,
image = canny_img,
guidance_scale = 6,
controlnet_conditioning_scale = 0.7,
num_inference_steps = 20
).images[0]

Can anyone help me?

[asomoza]

Hi, the error you're getting is because the model id or path you're using is wrong and we don't what they are:

MMIX_MODEL_PATH = os.getenv('MMIX_MODEL_PATH')
IP_ADAPTER_PATH = os.getenv('IP_ADAPTER')
CANNY_MODEL_PATH=os.getenv('CANNY_MODEL')

The recommendation here it to first use online models that you can verify they work and then use local ones, if the later ones fails like yours, that the problem is with your local files.

In your case that guide uses online models so you can start by using those and then change to local ones.

[Dmitr15]

Hi! Do you mean i need to run models through pipline from diffusers library?
I mean:
from transformers import pipeline

[asomoza]

no, I don't really understand what you mean because your example line is importing a pipeline from transformers and not diffusers.

I mean just with the online models that we can check and know that they're working, for example in the guide they use these ones:

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-canny", 
    torch_dtype=torch.float16,
    varient="fp16")
 
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "Yntec/AbsoluteReality",
    controlnet=controlnet, 
    torch_dtype=torch.float16)