Stable Diffusion
Stable Diffusion is a deep learning model which can be used to generate images primarily based on text descriptions.
It is also a very fast moving field and the information in this page may get outdated when you read this.
As an experimental feature, Visual Compositor supports integration with Stable Diffusion Web UI with Control Net (one module). After doing the initial setup, then Visual Compositor will be able to communicate with the Stable Diffusion Web UI via Stable Diffusion Txt2Img Node and Stable Diffusion Img2Img nodes, both in the editor and in runtime builds.
Initial Setup
- Ensure that your hardware meets the requirements as specified in the documentation of Stable Diffusion Web UI
- Get Stable Diffusion Web UI locally, either by doing
git clone
or downloading the latest version directly from the Releases page. - Modify the execution script to enable API calls.
On Windows for example, modifywebui-user.bat
and edit the line containingCOMMANDLINE_ARGS
to:
set COMMANDLINE_ARGS=--api --api-log
- Execute the execution script, i.e
webui-user.bat
on Windows. - Confirm that Stable Diffusion Web UI is running, for example by visiting http://localhost:7860/.
- (Optional) Install ControlNet by following these steps:
- Open the "Extensions" tab in the web UI
- Open "Install from URL" sub-tab.
- Enter
https://github.com/Mikubill/sd-webui-controlnet
in the "URL for extension's git repository" field. - Press "Install" button.
- Open "Installed" sub-tab.
- Make sure
sd-webui-controlnet
item is checked - Press "Apply and restart UI".
VisualCompositor only supports ControlNet with sdapiv12img compatible API starting from version
0.29.2-preview
.
Stable Diffusion Integration
After doing the initial setup once, then we can proceed to configuring Visual Compositor to use Stable Diffusion by following these steps:
Confirm that Stable Diffusion Web UI is running, for example by visiting http://localhost:7860/.
In the inspector of the VisualCompositor component, press the New button next to the Stable Diffusion Settings field to create a Stable Diffusion Settings asset.
Click the new Stable Diffusion Settings asset to open it in the inspector.
Confirm that the following properties have been initialized:
- SD Models
- Samplers
- Control Net Models (Only if Control Net extension has been setup for the Web UI).
If VisualCompositor can connect to Stable Diffusion Web UI, then the above properties will be automatically filled. Otherwise, a certain configuration error is preventing VisualCompositor to connect to it.
- Congratulations ! You should now be able to use Stable Diffusion nodes in Visual Compositor.
ControlNet Post API
Since VisualCompositor from version 0.29.2-preview
only supports ControlNet with sdapiv12img compatible API,
we need to ensure that the following fields in the Stable Diffusion Settings asset are set up correctly.
Fields | Correct Values |
---|---|
Post Control Net Txt 2 Img API | /sdapi/v1/txt2img |
Post Control Net Img 2 Img API | /sdapi/v1/img2img |
Default Property Values for Nodes
We can configure the default property values when creating new Stable Diffusion nodes by modifying the fields under Basic Parameters and Control Net Parameters.
Synchronizing Stable Diffusion Web UI to Unity
The buttons at the bottom of the inspector of Stable Diffusion Settings asset have the following functionalities:
Button | Functionality |
---|---|
Update SD Models | Refresh the registered Stable Diffusion models in Visual Compositor with the ones in Stable Diffusion Web UI, which is useful after adding or deleting models. |
Update Samplers | Refresh the registered samplers in Visual Compositor with the ones in Stable Diffusion Web UI. |
Update Control Net Models | Refresh the registered Control Net models in Visual Compositor with the ones in Stable Diffusion Web UI, which is useful after adding or deleting control net models. |
Get Active Model | Output the currently active Stable Diffusion model in the Console Window. |