Stable Diffusion

Stable Diffusion is a deep learning model which can be used to generate images primarily based on text descriptions.

It is also a very fast moving field and the information in this page may get outdated when you read this.

As an experimental feature, Visual Compositor supports integration with Stable Diffusion Web UI with Control Net (one module). After doing the initial setup, then Visual Compositor will be able to communicate with the Stable Diffusion Web UI via Stable Diffusion Txt2Img Node and Stable Diffusion Img2Img nodes, both in the editor and in runtime builds.

Initial Setup

  1. Ensure that your hardware meets the requirements as specified in the documentation of Stable Diffusion Web UI
  2. Get Stable Diffusion Web UI locally, either by doing git clone or downloading the latest version directly from the Releases page.
  3. Modify the execution script to enable API calls.
    On Windows for example, modify webui-user.bat and edit the line containing COMMANDLINE_ARGS to:
set COMMANDLINE_ARGS=--api --api-log
  1. Execute the execution script, i.e webui-user.bat on Windows.
  2. Confirm that Stable Diffusion Web UI is running, for example by visiting http://localhost:7860/.
  3. (Optional) Install ControlNet by following these steps:
    1. Open the "Extensions" tab in the web UI
    2. Open "Install from URL" sub-tab.
    3. Enter https://github.com/Mikubill/sd-webui-controlnet in the "URL for extension's git repository" field.
    4. Press "Install" button.
    5. Open "Installed" sub-tab.
    6. Make sure sd-webui-controlnet item is checked
    7. Press "Apply and restart UI".

VisualCompositor only supports ControlNet with sdapiv12img compatible API starting from version 0.29.2-preview.

Stable Diffusion Integration

After doing the initial setup once, then we can proceed to configuring Visual Compositor to use Stable Diffusion by following these steps:

  1. Confirm that Stable Diffusion Web UI is running, for example by visiting http://localhost:7860/.

  2. In the inspector of the VisualCompositor component, press the New button next to the Stable Diffusion Settings field to create a Stable Diffusion Settings asset.

  3. Click the new Stable Diffusion Settings asset to open it in the inspector.

  4. Confirm that the following properties have been initialized:

    1. SD Models
    2. Samplers
    3. Control Net Models (Only if Control Net extension has been setup for the Web UI).

If VisualCompositor can connect to Stable Diffusion Web UI, then the above properties will be automatically filled. Otherwise, a certain configuration error is preventing VisualCompositor to connect to it.

  1. Congratulations ! You should now be able to use Stable Diffusion nodes in Visual Compositor.

ControlNet Post API

Since VisualCompositor from version 0.29.2-preview only supports ControlNet with sdapiv12img compatible API, we need to ensure that the following fields in the Stable Diffusion Settings asset are set up correctly.

Fields Correct Values
Post Control Net Txt 2 Img API /sdapi/v1/txt2img
Post Control Net Img 2 Img API /sdapi/v1/img2img

Default Property Values for Nodes

We can configure the default property values when creating new Stable Diffusion nodes by modifying the fields under Basic Parameters and Control Net Parameters.

Synchronizing Stable Diffusion Web UI to Unity

The buttons at the bottom of the inspector of Stable Diffusion Settings asset have the following functionalities:

Button Functionality
Update SD Models Refresh the registered Stable Diffusion models in Visual Compositor with the ones in Stable Diffusion Web UI, which is useful after adding or deleting models.
Update Samplers Refresh the registered samplers in Visual Compositor with the ones in Stable Diffusion Web UI.
Update Control Net Models Refresh the registered Control Net models in Visual Compositor with the ones in Stable Diffusion Web UI, which is useful after adding or deleting control net models.
Get Active Model Output the currently active Stable Diffusion model in the Console Window.