Interface IWorker

The main interface to execute neural networks (a.k.a models). IWorker abstracts implementation details associated with various hardware devices (CPU, GPU and NPU in the future) that can execute neural networks and provides clean and simple interface to: 1) specify inputs, 2) schedule the work and 3) retrieve outputs. Internally IWorker translates description of the neural network provided by Model instance into the set of operations that are sent to hardware device for execution in a non-blocking (asynchronous) manner.

The following is a simple example of image classification using pretrained neural network:

    using UnityEngine;
    using Unity.Barracuda;

    public class ImageRecognitionSample : MonoBehaviour
    {
        // small ready to use image classification neural network in ONNX format can be obtained from https://github.com/onnx/models/tree/master/vision/classification/mobilenet
        public NNModel onnxAsset;
        public Texture2D imageToRecognise;

        private IWorker worker;
        void Start()
        {
            worker = onnxAsset.CreateWorker();
        }

        void Update()
        {
            // convert texture into Tensor of shape [1, imageToRecognise.height, imageToRecognise.width, 3]
            using (var input = new Tensor(imageToRecognise, channels:3))
            {
                // execute neural network with specific input and get results back
                var output = worker.Execute(input).PeekOutput();

                // the following line will access values of the output tensor causing the main thread to block until neural network execution is done
                var indexWithHighestProbability = output.ArgMax()[0];

                UnityEngine.Debug.Log($"Image was recognised as class number: {indexWithHighestProbability}");
            }
        }

        void OnDisable()
        {
            worker.Dispose();
        }
    }

The following example demonstrates the use of coroutine to continue smooth app execution while neural network executes in the background:

    using UnityEngine;
    using Unity.Barracuda;
    using System.Collections;
    public class CoroutineImageRecognitionSample : MonoBehaviour
    {
        // small ready to use image classification neural network in ONNX format can be obtained from https://github.com/onnx/models/tree/master/vision/classification/mobilenet
        public NNModel onnxAsset;
        public Texture2D imageToRecognise;

        private IWorker worker;
        void Start()
        {
            worker = onnxAsset.CreateWorker();
            StartCoroutine(ImageRecognitionCoroutine());
        }

        IEnumerator ImageRecognitionCoroutine()
        {
            while (true)
            {
                // convert texture into Tensor of shape [1, imageToRecognise.height, imageToRecognise.width, 3]
                using (var input = new Tensor(imageToRecognise, channels:3))
                {
                    // execute neural network with specific input and get results back
                    var output = worker.Execute(input).PeekOutput();

                    // allow main thread to run until neural network execution has finished
                    yield return new WaitForCompletion(output);

                    var indexWithHighestProbability = output.ArgMax()[0];
                    UnityEngine.Debug.Log($"Image was recognised as class number: {indexWithHighestProbability}");
                }

                // wait until a new image is provided
                var previousImage = imageToRecognise;
                while (imageToRecognise == previousImage)
                   yield return null;
            }
        }

        void OnDisable()
        {
            worker.Dispose();
        }
    }

Use WorkerFactory.CreateWorker or Model.CreateWorker to create new worker instance.

Namespace: Unity.Barracuda
Syntax
public interface IWorker : IDisposable

Properties

scheduleProgress

Reports the fraction (from 0.0 to 1.0) of the model that was scheduled for the execution since the last call to StartManualSchedule. This property will return 0.0 immediately after calling StartManualSchedule and will return 1.0 once the complete model was scheduled. This property will monotonuosly increase with the every iteration of IEnumerator that was obtained by calling StartManualSchedule.

Declaration
float scheduleProgress { get; }
Property Value
Type Description
Single

Methods

Execute()

Non-blocking API that schedules network execution in one go.

Declaration
IWorker Execute()
Returns
Type Description
IWorker

IWorker instance

Execute(IDictionary<String, Tensor>)

Non-blocking API that takes multiple input tensors and schedules network execution in one go.

Declaration
IWorker Execute(IDictionary<string, Tensor> inputs)
Parameters
Type Name Description
IDictionary<String, Tensor> inputs

input Tensor Dictionary: name -> Tensor
Returns
Type Description
IWorker

IWorker instance

Execute(Tensor)

Non-blocking API that takes single input tensor and schedules network execution in one go. Useful when network have only one input as input name is not needed.

Declaration
IWorker Execute(Tensor input)
Parameters
Type Name Description
Tensor input

input Tensor
Returns
Type Description
IWorker

IWorker instance

FlushSchedule(Boolean)

Non-blocking API that starts immediate execution on the part of the network that was scheduled so far. Optional blocking flag can force this function to block until execution is complete.

Declaration
void FlushSchedule(bool blocking = false)
Parameters
Type Name Description
Boolean blocking

if blocking True, wait for completion

PeekConstants(String)

Returns references to constants tensors for a layer. This reference might be valid only until the next Execute() or Dispose() method is called on the worker. IMPORTANT: if you want tensor to outlive the worker, use CopyOutput() method or follow with TakeOwnership() call on the tensor, also worker Execute() or PrepareForInput() should have been called at least once for the tensors to exist.

Declaration
Tensor[] PeekConstants(string layerName)
Parameters
Type Name Description
String layerName

Layer name
Returns
Type Description
Tensor[]

array of constant Tensors

PeekOutput()

Non-blocking API that returns a reference to the main output tensor. This reference will be valid only until the next Execute() or Dispose() method is called on the worker. Useful when network has only one output. IMPORTANT: if you want tensor to outlive the worker, use CopyOutput() method or follow with TakeOwnership() call on the tensor.

Declaration
Tensor PeekOutput()
Returns
Type Description
Tensor

output Tensor

PeekOutput(String)

Non-blocking API that returns a reference to output tensor by specified name. This reference will be valid only until the next Execute() or Dispose() method is called on the worker. IMPORTANT: if you want tensor to outlive the worker, use CopyOutput() method or follow with TakeOwnership() call on the tensor.

Declaration
Tensor PeekOutput(string name)
Parameters
Type Name Description
String name

output name
Returns
Type Description
Tensor

output Tensor

PrepareForInput(IDictionary<String, TensorShape>)

Optional API to prepare network execution for inputs of particular shapes. Useful to initialize execution device ahead of the first call to Execute.

Declaration
void PrepareForInput(IDictionary<string, TensorShape> inputShapes)
Parameters
Type Name Description
IDictionary<String, TensorShape> inputShapes

Dictionary of tensor name -> input shapes

SetInput(String, Tensor)

Assign tensor x to the named input of the network. String name specifies the name of the input.

Declaration
void SetInput(string name, Tensor x)
Parameters
Type Name Description
String name

Tensor name
Tensor x

Tensor

SetInput(Tensor)

Specify single tensor x as the only input for the network. Useful when network has only one input and caller does not need to specify input's name.

Declaration
void SetInput(Tensor x)
Parameters
Type Name Description
Tensor x

input Tensor

StartManualSchedule()

Non-blocking API that allows manual scheduling of the model one layer at the time. Call MoveNext on the IEnumerator obtained from calling this function to schedule next layer of the model.

Declaration
IEnumerator StartManualSchedule()
Returns
Type Description
IEnumerator

Manual schedule iterator

StartManualSchedule(IDictionary<String, Tensor>)

Non-blocking API that takes mutliple input tensors and schedules network execution one layer at the time. Call MoveNext on the IEnumerator obtained from calling this function to schedule next layer of the model.

Declaration
IEnumerator StartManualSchedule(IDictionary<string, Tensor> inputs)
Parameters
Type Name Description
IDictionary<String, Tensor> inputs

input Tensor Dictionary: name -> Tensor
Returns
Type Description
IEnumerator

Manual schedule iterator

StartManualSchedule(Tensor)

Non-blocking API that takes single input tensor and schedules network execution one layer at the time. Call MoveNext on the IEnumerator obtained from calling this function to schedule next layer of the model.

Declaration
IEnumerator StartManualSchedule(Tensor input)
Parameters
Type Name Description
Tensor input

input Tensor
Returns
Type Description
IEnumerator

Manual schedule iterator

Summary()

Returns a string summary after execution.

Declaration
string Summary()
Returns
Type Description
String

string summary after execution

Extension Methods