Advice on style training

[Clement-Lelievre]

Hi all,

I'm looking for advice on how to best train a LoRA on the following task: the LoRA should learn to represent black and white wireframe drawings seen from top (orthogonal view ie objects seen from top, no perspective).
I attach one example of image from my dataset depicting a desk in the above mentioned style.

So far I've mostly tried fine-tuning flux dev, varying lr, rank etc. to see what I can get; and the LoRA partially learns but definitely isn't great.

Do you have any recommendations of model/hyperparams? (if this task is feasible at all)

Thanks vm!
Happy to provide more details if needed

(maybe @asomoza ?)

[asomoza]

can you post an image generated with your lora?

It's probably doable with any model but it depends on your definition of 'isn't great'. For example, there's a lora for ikea instructions trained in the base SDXL model which is fine if you don't consider the text.

About your decision to use Flux, I haven't tried to train a lora on it yet but for what I've seen and tested, Flux looses a lot of the quality when you finetune it, the loras I tested work fine but they degrade a lot the other stuff and also the model isn't that great for lines (for example anime) and it's really overtrained in professional photos with bokeh.

For example, take this lora for SDXL and this one for Flux which are for 3D wireframes, if you ask me, based in the demo images, I think the SDXL one is better.

Also how big if your dataset? Depending on the size, you will probably need to finetune a model that's really over trained in lines like the anime models. This probably needs a bigger dataset than a normal character lora.

I really can't help with the hyperparameters for Flux, but one good start would be to train the whole model and then extract loras in different ranks to test which one works better for your use case. Also as a good default start, you should use the same LR than what the base model used.

[asomoza]

I misunderstood what you said, I read it again and you were referring to the base training, don't really know about Flux but SDXL can do top down views without a problem if you fine tune it for it.

I did a quick search for a lora and I could generate these images really easy:

Didn't see anything similar in Flux, just some people that trained some top down map generators.

What you're trying to do shouldn't be that hard, you're just training with grayscale images and with straight lines, a lot easier than some of the other finetuning's like those weird styles like yarn or spaghetti, also I saw someone that trained a lora with dotted lines for drawing which seems a lot more complex to train than yours.

My guess is that since you're fixated in using Flux, you will need to teach it everything, first the top view, then the line drawings and then the conversion of the objects between for example, a front view and a top view.

Probably you won't be able to do this with dreambooth and will need to train a full lora or model for this, also take into account, that people mostly finetune loras with Flux for portrait photos of a person with a very simple pose, that is probably for a reason.

[Clement-Lelievre]

thanks for the reply. haha don't worry I'm not fixated with using flux, if another model is better suited to the task then I'll go for it.
So you're saying you found an existing SDXL LoRA that generated the top view images you shared?
I will look into dreambooth sdxl lora training

[asomoza]

yeah, this one but as I told you, it's for a model that's not that safe to use but that is only trained with lines.

I used to never look at those models but I read and then tested (and corroborted) that they're really good for a lot of other stuff because of the same training. The problem is that even if you don't prompt it, you can sometimes get some nsfw results, so if that bothers you, you will need to train or find a model trained with lines that is sfw but because of the same, they lack the understanding of these newer models.

Same problem as base models with anatomy, SDXL, SD3, an all the newer models are really bad with anatomy because they weren't trained with "good anatomy" to avoid the backslash and to be stigmatized as nsfw.

[Clement-Lelievre]

update: I fine-tuned SDXL (still with Dreambooth LoRA, using this diffusers script) with the default params, and while the LoRA quickly learnt the style (white background, thin dark lines, wireframe style) it struggles with learning orthogonality, specifically drawing an object as seen from top (like the first image I posted, which comes fro my training dataset). Flux struggled with this too.
I tried to play with the hyperparams a bit but got no improvement.
Not sure if it is realistic to teach this "concept" (I guess it's a concept?) of orthogonality in a Dreambooth LoRA fine-tuning, I don't have enough experience of it.

[asomoza]

yeah, this probably is not something you can teach the model with just dreambooth and more like a full lora or finetune.

I also have an update, I've seen a lot more people producing Flux loras lately (probably because of Flux Gym and there's been a couple of new ones that are using styles which seems good, at least in the demo images, so this makes me think that you will be able to train your idea with Flux.

Probably same as you have guessed already, not with dreambooth and with a larger dataset.

[Clement-Lelievre]

hey @asomoza

I wanted to update you in case you're interested: I've come up with a way to remove one constraint in my training (the orthographic representation). So now, I am trying to teach the model the style shown on the image I displayed in my first message above. I have a balanced mix of all views in my dataset (diverse objects, shown in front, side or top view).

(I will then load these lora weights and use a controlnet in order to get this style AND following the structure of a depth map, but that's another story)

I couldn't make flux learn it, now trying with SDXL. Using this script with hyperparams defaults for now. As you can see, the training is happening to some degree, as validation inferences show a drawing style, and dark thick edges, but their style is far from the wireframe, minimalist drawing style of the dataset