Recorded Testing

Recorded Playbacks (recordings) can be used to "drive" a Unity Test, combining the simplicity of recorded playback creation with the flexibility and power of the Unity Test Framework.

Automatic Recorded Test Creation

Once you have one or more recordings, use the Test Generation feature to quickly create Unity Tests that validate the proper playback of a recording. To Generate tests from recordings, navigate to the Test Generation section of the Automated QA Hub or press the three dots to the left of recordings in the recorded playback window..

Managing & Running Recorded Tests

Recorded Tests, like all Unity Tests, can be managed and run from the Unity Test Runner: navigate to Window > General > Test Runner and then select the PlayMode tab. From the Test Runner UI, Unity Tests can be run in the Unity Editor or on a local device (e.g. on a phone or tablet).

Running Recorded Tests on a Cloud Device Farm (Access Controlled while in Development)

Run Recorded Tests (and other Unity Tests) on iOS and Android devices managed by Unity.

[Required for Android Only] Configure your Project for Cloud Testing on Android Devices

  1. Open your project in the Unity Editor with Android as the build target
  2. Navigate to File > Build Settings and make sure Build App Bundle (Google Play) is not checked.
  3. Add a custom Android Manifest to your project to bypass any device permission dialogs that may occur during testing. Navigate to Edit > Project Settings... > Player > Publishing Settings and ensure Custom Main Manifest is checked.

Replace the source of the Android Manifest (/Assets/Plugins/Android/AndroidManifest.xml) with the below:

<?xml version="1.0" encoding="utf-8"?>
<!-- GENERATED BY UNITY. REMOVE THIS COMMENT TO PREVENT OVERWRITING WHEN EXPORTING AGAIN-->
<manifest
    xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.unity3d.player"
    xmlns:tools="http://schemas.android.com/tools">
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
    <application>
        <activity android:name="com.unity3d.player.UnityPlayerActivity"
                  android:theme="@style/UnityThemeSelector">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
            <meta-data android:name="unityplayer.UnityActivity" android:value="true" />
            <meta-data android:name="unityplayer.SkipPermissionsDialog" android:value="true" />
        </activity>
    </application>
</manifest>

Run Tests in the Cloud

Once you have one or more Recorded Tests (or any Unity Tests) that you'd like to run on a cloud iOS or Android device:

  1. Designate the Unity Tests that you would like to run in the cloud by adding [CloudTest] above the test definition. Note: this is done by default for any tests automatically generated from recorded playback.
  2. Choose a build for your run by navigating to Automated QA Hub > Cloud Testing
    1. Use a previously uploaded build with the drop down menu for builds
    2. Make a new build by pressing "Build & Upload". This will create a build bundled with all of your tests tagged with [CloudTest] and upload it to our cloud testing service.
  3. If you'd like to choose which devices to run your test on then create a Cloud Test Device Input Asset from Assets > Create > Automated QA > Cloud Test Device Input.
  4. You can use https://awsdevicefarm.info/ as a reference to populate this file.
    • Only one device OS Version can be specified per job and it should match the OS column.
    • Device Names should be exact matches with the Name column except for iOS devices where it should drop the word Apple.
    • Not all devices that show up on the list are available please use the validate button to check if the device is truly available.
  5. Once you've created your Cloud Test Device Input hit validate while logged in to your unity account to see a debug statement in the console which will show matching devices for your criteria.
  6. Run a cloud test job
    1. Once you've added your Cloud Test Device Input asset into the Cloud Test Runner window or left it blank to use default devices you can click run on Device Farm.
    2. The Job ID should be automatically populated in the jobid field. Press "Refresh" to check the status of your tests. Cloud tests run sequentially and in real time (i.e. timescale 1).
    3. Press "Get Test Results" to view the results of your tests in an HTML file.
  7. View a job's results by selecting a past job from the dropdown and pressing "Get Test Results" while the jobId field is empty.

Supported Devices

By default, we run tests on 5 of randomly selected iOS or Android devices (depending on your build target).

Advanced Features

Manual Recorded Test Creation

Recorded Playbacks can be used "drive" any Play Mode UnityTest. To manually create a test driven by a recording, create a Unity Test and then add [RecordedTest("[relative path of recording (relative to /Assets)]")] right after the test's [UnityTest] attribute in the source code of the test.

For example, to validate that a tutorial loads as expected:

  1. Create a recorded playback that ends with the tutorial loading
  2. Rename the resulting recording-[timestamp].json file to something more descriptive like "load_tutorial.json"
  3. Create a new Unity Test with [RecordedTest("Recordings/load_tutorial.json")] below the [UnityTest] attribute
  4. Inside the Unity Test: a. Load the scene where your recording begins, e.g. SceneManager.LoadScene("0_Title"); b. Once playback is complete or some period of time has passed, Assert that some key condition is true

Example RecordedTest usage:

namespace Tests
{
    public class SmokeTests : RecordedTestSuite
    {
        [UnityTest]
        [RecordedTest("Recordings/load_tutorial.json.json")]
        public IEnumerator validate_tutorial_loads()
        {
           SceneManager.LoadScene("0_Title");
           ...

           //wait until the Tutorial scene loads
           while (SceneManager.GetActiveScene().name != "Tutorial")
           {
               yield return null;
           }

           Assert.NotNull(GameObject.Find("Tutorial_Manager");
        }

    }
}