Recorded Testing

Recorded Playbacks (recordings) can be used to "drive" a Unity Test, combining the simplicity of recorded playback creation with the flexibility and power of the Unity Test Framework.

Automatic Recorded Test Creation

Once you have one or more recordings, use the Test Generation feature to quickly create Unity Tests that validate the proper playback of a recording. To Generate tests for each recording in the Assets/Recordings directory, navigate to Tools > Automated QA > Generate Recorded Tests (from the Unity menu bar).

Managing & Running Recorded Tests

Recorded Tests, like all Unity Tests, can be managed and run from the Unity Test Runner: navigate to Window > General > Test Runner and then select the PlayMode tab. From the Test Runner UI, Unity Tests can be run in the Unity Editor or on a local device (e.g. on a phone or tablet).

[Early Access] Running Recorded Tests on a Cloud Device Farm

This package includes support for running Recorded Tests (and other Unity Tests) on Android Phones hosted on AWS Device Farm - no AWS account needed.

Through June 2021, we’re offering 100 hours of cloud testing free to every organization with a Unity Pro license. After the free trial, cloud testing will cost $10 per device hour. Please email us at a AutomatedQA@unity3d.com for access!

Configure your Project for Cloud Testing

  1. Open your project in the Unity Editor with Android as the build target
  2. Navigate to File > Build Settings and make sure Build App Bundle (Google Play) is not checked.
  3. Add a custom Android Manifest to your project to bypass any device permission dialogs that may occur during testing. Navigate to Edit > Project Settings... > Player > Publishing Settings and ensure Custom Main Manifest is checked.

Replace the source of the Android Manifest (/Assets/Plugins/Android/AndroidManifest.xml) with the below:

<?xml version="1.0" encoding="utf-8"?>
<!-- GENERATED BY UNITY. REMOVE THIS COMMENT TO PREVENT OVERWRITING WHEN EXPORTING AGAIN-->
<manifest
    xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.unity3d.player"
    xmlns:tools="http://schemas.android.com/tools">
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
    <application>
        <activity android:name="com.unity3d.player.UnityPlayerActivity"
                  android:theme="@style/UnityThemeSelector">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
            <meta-data android:name="unityplayer.UnityActivity" android:value="true" />
            <meta-data android:name="unityplayer.SkipPermissionsDialog" android:value="true" />
        </activity>
    </application>
</manifest>

Run Tests in the Cloud

Once you have one or more Recorded Tests (or any Unity Tests) that you'd like to run on a cloud Android device:

  1. [If you're running any Recorded Tests] Navigate to Tools > Automated QA > Recording Upload Window and then press "Upload All"
  2. Navigate to Tools > Automated QA > Cloud Test Runner Window and then press "Build & Upload". This will create a build bundled with all of your play mode tests and upload it to our cloud testing service.
  3. The Build ID should be automatically populated in the buildid field. Press "Run on Device Farm".
  4. The Job ID should be automatically populated in the jobid field. Press "Refresh" to check the status of your tests. Cloud tests usually take 15-30 minutes to complete.
  5. Press "Get Test Results" to view the results of your tests in an HTML file.

Supported Devices

Currently, we run tests by default on 5 of the most popular Android devices:

  • Samsung Galaxy S9 (OS 9)
  • Samsung Galaxy S20 (OS 10)
  • Samsung Galaxy Note 10 (OS 9),
  • Google Pixel 4 XL (OS 10)
  • Google Pixel XL (OS 8.0.0).

Please reach out to us at AutomatedQA@unity3d.com if you would like to run tests on other devices offered by AWS Device Farm.

Advanced Features

Manual Recorded Test Creation

Recorded Playbacks can be used "drive" any Play Mode UnityTest. To manually create a test driven by a recording, create a Unity Test and then add [RecordedTest("[relative path of recording (relative to /Assets)]")] right after the test's [UnityTest] attribute in the source code of the test.

For example, to validate that a tutorial loads as expected:

  1. Create a recorded playback that ends with the tutorial loading
  2. Rename the resulting recording-[timestamp].json file to something more descriptive like "load_tutorial.json"
  3. Create a new Unity Test with [RecordedTest("Recordings/load_tutorial.json")] below the [UnityTest] attribute
  4. Inside the Unity Test: a. Load the scene where your recording begins, e.g. SceneManager.LoadScene("0_Title"); b. Once playback is complete or some period of time has passed, Assert that some key condition is true

Example RecordedTest usage:

namespace Tests
{
    public class SmokeTests : RecordedTestSuite
    {
        [UnityTest]
        [RecordedTest("Recordings/load_tutorial.json.json")]
        public IEnumerator validate_tutorial_loads()
        {
           SceneManager.LoadScene("0_Title");
           ...

           //wait until the Tutorial scene loads
           while (SceneManager.GetActiveScene().name != "Tutorial")
           {
               yield return null;
           }

           Assert.NotNull(GameObject.Find("Tutorial_Manager");
        }

    }
}