Language:
Page Info
Tags:
Engine Version:

Automation System Overview

Choose your OS:

The Automation System is built on top of the Functional Testing Framework, which is designed to do gameplay level testing, which works by performing one or more automated tests. Most tests that are written will be functional tests, low-level core or Editor tests that need to be written for using the Automation Framework system. These tests that are written can be broken down into the following categories depending on their purpose or function:

Test Type

spacer.png

Description

Unit

API level verification tests. See TimespanTest.cpp or DateTimeTest.cpp for examples of these.

Feature

System-level verification tests such as verify PIE works, verify in-game stats works and very changing resolution works. See EditorAutomationTests.cpp or EngineAutomationTests.cpp for examples of these.

Smoke

Smoke tests are just considered a speed promise by the implementer. They are intended to be fast so they can run everytime the Editor, game, or commandlet starts. They are also selected default in the UI .

All Smoke tests are intended to complete within 1 second. Only mark Unit Tests or fast Feature Tests as Smoke Tests.

Content Stress

More thorough testing of a particular system to avoid crashes, such as loading all maps or loading and compiling all Blueprints. See EditorAutomationTests.cpp or EngineAutomationTests.cpp for examples of these.

Screenshot Comparison

This enables your QA testing to quickly compare screenshots to identify potential rendering issues between versions or builds.

Automation Tests moved to Plugins

For a long time, automation tests were spread across the Engine and Editor which meant that when you shipped your title, these were included. These have now been moved to their own plugins that can be individually enabled. This also means that since these tests are located in plugins, you can choose to include them with your packaged builds when you compile. Plugins also can store content so that it no longer has to be kept in the Engine Content folder.

Because of this change, the type of test being created will dictate where it should be stored. Use the table below to look at the test type and which plugin it should be stored with:

Test Type

Plugin to store it in

Unit

These can continue to be stored in the same module as the code.

Project Agnostic Runtime Tests

RuntimeTests Plugin

Project Agnostic Editor Tests

EditorTests Plugin

Functional Tests

EngineTest Game folder

Some tests are still in the Engine that has yet to be moved to a Plugin. These will be moved out over time to corresponding Plugins that can be found in the Plugins Browser listed under Testing.

Enabling Automation Test Plugins

To enable the plugins for the different tests available, go to the file menu and select Window > Plugins to open the Plugin Browser Window.

EnablePlugins.png

In the left panel, select the Testing category and enable the test plugins you want to use.

Click image for full size.

Once you've selected these, you will be prompted to restart the Editor. Click the Restart Now button.

RestartTheEditor.png

Test Design Guidelines

When you're testing your games or projects, the following are some general guidelines that Epic follows for its automation testing:

  • Don't assume the state of the game or the Editor. Tests can be run out of order or parallel across machines.

  • Leave the state of the files on disk the way you found them. If a test generates a file, delete it when the test is done. (This may change in the future to opt into a way to automatically delete these types of generated files).

  • Assume the test was left in a bad state the last time it was run. A good habit would be to generate files to attempt to delete them before the test starts.

Running Automation Tests

  1. Open any project.

  2. Enable the Plugins available to use for testing and restart the Editor.

  3. In the Editor, go to Window > Test Automation.

    Click image for full size.

    For this menu option to be visible, you must first enable at least one automation tests plugin .

  4. In the Automation tab of the Sessions Frontend under the Test Name column, enable the following:

    Click image for full size.

    • Editor

    • Project

    • System

  5. In the Automation tab toolbar, click the Start Tests button.

    Click image for full size.

    As the tests complete, you can follow the progress in the Test Name window.

    Click image for full size.

Essentials

The Automation System provides the ability to perform Unit Testing, Feature Testing, and Content Stress Testing using the power of the Unreal Message Bus to increase stability.