Understand models in Inference Engine

Inference Engine can import and run trained machine learning model files in Open Neural Network Exchange (ONNX) format.

To get a model that's compatible with Inference Engine, you can do one of the following:

How Inference Engine optimizes a model

When you import a model, each ONNX operator in the model graph becomes an Inference Engine layer.

To check the list of layers in the imported model, in the order Inference Engine runs them, open the Model Asset Inspector. For more information, refer to Supported ONNX operators.

Inference Engine optimizes models to make them smaller and more efficient. For example, Inference Engine might do the following to an imported model:

  • Remove a layer or subgraph and turn it into a constant.
  • Replace a layer or subgraph with a simpler layer or subgraph that works the same way.
  • Set a layer to run on the CPU, if the data must be read at inference time.

The optimization doesn't affect what the model inputs or outputs.

Model inputs

You can get the shape of your model inputs in one of two ways:

  • Inspect a model to use the [inputs](xref:Unity.Inference Engine.Model.inputs) property of the runtime model.
  • Select your model from the Project window to open the Model Asset Inspector and view the Inputs section.

The shape of a model input consists of multiple dimensions, defined as a [DynamicTensorShape](xref:Unity.Inference Engine.DynamicTensorShape).

The dimensions of a model input are either static or dynamic:

  • An int denotes a static dimension.
  • A named string (for example, batch_size, height) represents a dynamic dimension. These names come from the ONNX model.

Static dimensions

The value of the int defines the specific shape of the input the model accepts.

For example, if the Inputs section displays (1, 1, 28, 28), the model only accepts an input tensor of shape 1 x 1 x 28 x 28.

Dynamic dimensions

A dynamic dimension allows flexibility in input shapes. When the shape of a model input contains a named dynamic dimension, for example batch_size or height, that input dimension can be any size.

For example, if the input has a shape (batch_size, 1, 28, 28), the first dimension of the input shape can be any size.

When you define the input tensor for this input shape, the following tensor shapes are valid:

[1, 1, 28, 28]
[2, 1, 28, 28]
[3, 1, 28, 28] ...

If you change the size of another dimension, however, the input tensor isn't valid. For example:

[1, 3, 28, 28]
Note

If a model uses inputs with dynamic input dimensions, Inference Engine might not be able to optimize the model as efficiently as a model that uses static input dimensions. This might slow down the model.

For more information on how to set a static value for a dynamic dimension, refer to Import settings for ONNX models.

Additional resources