Stage Monitor

Stage Monitor reports events from multiple instances of Unreal Engine

Windows
MacOS
Linux

When operating a stage in a live environment, multiple machines running instances of Unreal Engine (UE) work in tandem and rely on each other. Operators may use some instances to render on a LED wall, some could be used to modify the scene in the Editor, and others may be used for compositing. With Stage Monitor, you can receive reports of different events from all these instances of Unreal Engine and be able to troubleshoot any hiccups in the setup.

Stage_StageMonitorWindow_RAW.PNG

At the end of this page, you will be familiar with using Stage Monitor as well as adding custom events for your project.

Required Plugins

Add the following plugins to your project to get started with Stage Monitor.

Plugin Name

Purpose

Stage Monitoring

Main plugin enabling data providers and stage monitoring logic.

Virtual Production Utilities

Enabled with StageMonitoring plugin to be able to use VirtualProduction role's feature.

Switchboard

Switchboard is optional but it facilitates launching multiple instances of Unreal operating a stage. It can also be used to configure the role assigned to each instance.

Follow these steps to enable the plugins for your project:

  1. From the main menu in the Unreal Editor, select Edit > Plugins.

    Stage_PluginsMenu.png

  2. In the Plugins window, find the Stage monitor plugin and check Enable.

    Stage_PluginsFindStage.png

  3. When prompted, select Yes.

    Stage_BetaOptIn.png

  4. Click Restart Now to close the Editor and restart with your new Plugin activated.

    Stage_PluginReset_RAW.PNG

  5. Repeat the above process to enable the Virtual Production Utilities and Switchboard plugins.

These plugins may already be enabled by default, depending on the version of UE that you are working in. Either way, it's best to double check that they're enabled while in the Plugins menu.

Setting up Timecode

Each instance of Unreal Engine generates its own timecode. With multiple machines communicating together, events can appear to be out of order in Stage Monitor if they're not synchronized. To effectively monitor UE instances, enable Timecode on all instances being monitored and use the same timecode provider. See Timecode and Genlock for more information on synchronizing timecode across machines.

Assigning Machine Roles

When operating a stage, each machine can perform a different role, such as renderer, editor, or compositor. Instances of UE can be assigned roles in a command line argument or in Switchboard to utilize different logic based on their function. Stage Monitor uses these roles to determine what events an instance can generate as well as whether the instance should be monitored.

Assigning Roles in Switchboard

Switchboard will look for the file VPRoles.ini in each instance's project folder at \Config\Tags\VPRoles.ini. In the following example, the roles Render, Editor, and Compositor were added to the VPRoles.ini file:

[/Script/GameplayTags.GameplayTagsList]
GameplayTagList=(Tag="Render", DevComment="")
GameplayTagList=(Tag="Editor", DevComment="")
GameplayTagList=(Tag="Compositor", DevComment="")

In Switchboard, you can adjust the roles for each instance in the settings.

Stage_SwitchboardSettings.png

Assigning Roles in Command Line

You can assign each instance a role through a command line argument. When launching the instance of Unreal, add the argument : -VPRole=RoleName, where RoleName is the role you're assigning to the machine, such as Editor.

Opening Stage Monitor

Stage Monitor data providers are started by default unless role filtering is used. When each instance is launched, they will look for a Stage Monitor, and once found they will start sending events. Periodically, an instance that is monitoring others will broadcast messages to discover data providers. When a monitor is found, the data provider will then start to send out any events that are emitted. Launch Stage Monitor to confirm that information is being sent and received in the expected order.

Follow these steps to launch Stage Monitor:

  1. Expand the Window tab and under General > Developer Tools click Stage monitor.

    Stage_WindowTab_RAW.PNG

  2. Activate the Stage Monitor using the toggle switch next to Monitor Status.

    Stage_StageMonitorActivation_RAW.PNG

  3. Once activated, you should be able to view connected instances in the top menu.

    Stage_StageMonitorWindowInstances.png

  4. Additionally, once activated you should see a steady stream of stage events populating the lower window.

    Stage_StageMonitorWindowEvents.png

Using Stage Monitor

Once the information starts flowing into Stage Monitor, it can be a lot to process. Here are some items to be on the lookout for, and some tips to help you manage the event logs.

Critical State

Take Recorder uses the event type Critical State to designate when the scene is being recorded. While you're recording, any event could impact the result. This means that when a take has started, the stage enters Critical State, and when the take has stopped, the stage exits Critical State. Since Critical State is an event type, you can set your own Critical States for any scenario.

Follow these steps to see Stage Monitor enter Critical State with TakeRecorder.

  1. Launch Stage Monitor and confirm that all instances of UE are connected and properly reporting events.

  2. Launch Take Recorder by going to Windows > Cinematics > Take Recorder.

    Stage_LaunchTakeRecorder_RAW.PNG

  3. In the Take Recorder click the Start Take button.

    Stage_StartTake.png

  4. After a brief pause, you will notice the Stage Monitor status indicator turn from Green (not operational) to Red (operational) indicating that the Stage Monitor is now in the Critical State.

    Stage_StatusChange.png

  5. Notice that while in the Critical State events in the Event Log will be highlighted.

    Stage_CriticalEvents.png

  6. Click the Stop Take button in the Take Recorder to stop the take.

    Stage_StopTake.png

The video below shows this process in action.

Filtering

After a while, message logs become bigger and it's harder to find something specific within them. Using the filtering menu, you can easily focus on the message types you are interested in, data providers that you want to isolate, and to only see events that happened while the stage was operational.

  1. Launch the Stage Monitor by going to Window > Developer Tools > Miscellaneous.

    Stage_WindowTab_RAW.PNG

  2. Under the Event Log expand the Filters dropdown menu.

    Stage_FiltersMenu.png

  3. Expand the Message Type menu to filter by specific message types.

    Stage_FilterMessageType_RAW.PNG

  4. Expand the Provider menu to filter events by Event Provider.

    Stage_FilterProvider_RAW.PNG

  5. Select and expand the Critical State Source option to filter based on Critical State sources.

    Stage_FilterCriticalSource_RAW.PNG

Exporting / Importing

Once a session is done, it's possible to export all events and providers to a JSON file. You can then review these logs externally, or directly in UE.

  1. Launch the Stage Monitor by choosing Window > Developer Tools > Miscellaneous > Stage Monitor.

    Stage_WindowTab_RAW.PNG

  2. Near the top-left of the Stage Monitor notice the series of five buttons.

    Stage_AllButtons.png

  3. Click the disk icon to export the current log. After clicking the button you will be asked where you would like to save the log.

    Stage_ExportButton.png

  4. Click the orange dot on the left to enter review mode.

    Stage_ReviewModeButton.png

  5. When entering review mode, you will see the name next to CurrentSession change to reflect the file you are reviewing.

    Stage_SessionStatusChange.png

  6. While in review mode, click the folder icon to load a previously recorded log and review it.

    Stage_ReviewModeFolder.png

    Stage_LoadingLog_RAW.PNG

Once a file has been opened, you can use the same filters to isolate specific events.

Stage Monitor Event Types

A couple of types of Stage Monitor events are triggered with the base version of UE. New types can be added through C++ by adding a dependency to the StageDataCore module, and inheriting FStageProviderEventMessage for events. Additionally, you can extend FStageProviderPeriodicMessage for repetitive messages. Below is a table listing common events, their file location, and their purpose.

Event Name

File Location

Description

Critical State Provider Message

StageMessages.h

Sent when entering or exiting Critical State.

Stage Provider Discovery Response Message

StageMessages.h

Part of the discovery protocol between monitors and providers.

Stage Provider Close Message

StageMessages.h

Sent when a provider is shutting down.

Frame Performance Provider Message

StageMonitorUtils.h

Periodically sent to refresh provider's frame data.

Hitch Detection Message

FramePerformanceProvider.h

Sent when a frame takes more time than the maximum allowed time desired.

Genlock State Event

GenlockWatchdog.h

Sent when CustomTimeStep is a genlocked one and its stage changed.

Genlock Hitch Event

GenlockWatchdog.h

Sent when CustomTimeStep is a genlocked one and a genlock signal (a frame) was missed.

Timecode Provider State Event

TimecodeProviderWatchdog.h

Sent when state of TimecodeProvider changes.

Timed Data Monitor Channel ConnectionState Event

TimecodeProviderWatchdog.h

Requires TimedDataMonitor plugin. Sent when a channel connection state changes.

Timed Data Monitor Channel Evaluation State Event

TimedDataMonitorSubsystem.h

Requires TimedDataMonitor plugin. Sent when a channel evaluation state changes.

Adding Event Types

Follow these steps to add custom Stage Monitor event types.

  1. In the Editor's main menu, choose File > Open Visual Studio. If you don't see this option, select New C++ Class...

  2. In Visual Studio's Solution Explorer find the module, defined in a .Build.cs file, where you want to emit stage events. Append StageDataCore to the PublicDependencyModuleNames range:

    PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "StageDataCore" });

  3. In a C++ header file, define your stage event data structure and inherit from either FStageProviderEventMessage or FStageProviderPeriodicMessage . In the following header file MyStageEvents.h, FMyStageEvent is inheriting from FStageProviderEventMessage and expects the data to be int32:

    #include "StageMessages.h"
    
    #include "MyStageEvents.generated.h"
    
    USTRUCT ()
    struct FMyStageEvent : public FStageProviderEventMessage
    {
    GENERATED_BODY ()
    
    public:
    FMyStageEvent () = default;
    FMyStageEvent (int32 InMyVariable) : MyVariable (InMyVariable)
    {}
    
    public:
    int32 MyVariable = 0;
    }
  4. In your code, whenever you want to emit an event, call IStageDataProvider::SendMessage<>() with your event as the template parameter. For example, emit the event created above with the following::

    int32 DataToSend = 3;
    IStageDataProvider::SendMessage<FMyStageEvent>(EStageMessageFlags::Reliable, DataToSend);

Since UE sends messages using UDP, a message could be lost or dropped and never received. Using the flag EStageMessageFlags::Reliable ensures that the Stage Monitor will receive the message.

If you are sending a periodic message, you can use EStageMessageFlags::None instead of the Reliable flag.

Stage Monitor Settings

You can find settings for Stage Monitor either directly within the tool, or under your project's Project Settings.

  1. To access Stage Monitor settings from the Unreal Editor, go to Edit > Project Settings and Search Stage.

    Stage_OpenProjectSettings_RAW.PNG

    Stage_SearchProjectSettings.png

  2. To access Stage Monitor settings from within the tool, start by launching the Stage Monitor by going to Window > Developer Tools > Miscellaneous.

    Stage_WindowTab_RAW.PNG

  3. Once in the tool click the cogs button.

    Stage_SettingsButton.png

Below is a review of the most important settings and their purposes.

General Settings

Stage_PluginGeneralSettings.png

Setting Name

Description

Use Session ID

You can use a sessionId to differentiate between two stages operating on the same network. You can then pass that sessionId using the command line to each instance using -StageSessionId=.

Timeout Interval

Time required for a provider to consider a monitor to be disconnected as well as the time for a monitor to consider a provider to be disconnected.

Monitor Settings

Stage_PluginMonitorSettings.png

Setting Name

Description

Supported Roles

Used to filter monitoring logic to only be used when an instance has a specific role. Useful when you auto start monitoring but don't want it to be used on render nodes for example.

Auto Start

By default, monitoring logic doesn't start automatically. You can start it in the monitor panel but if you want it to be started when an instance is launched, you can check that setting. If you are not using role filtering, this means that any instance will be monitoring, which can increase network traffic a lot in case of multiple instances being used.

Provider Settings

Stage_PluginProviderSettings.png

Setting Name

Description

Supported Roles

Only enable data provider logic when an instance has a role from that list.

Message Type Role Exclusion

By default, any data provider will emit any event that is triggered. This can clutter monitoring logs a lot especially when hitch detection is enabled. For example, when using the editor, for sure some hitches will be triggered when the user goes in menus. To filter that, you can use this setting to assign specific roles for a given event type to filter which kind of instance can trigger that event.

Update Interval

UE periodically sends this message from all data providers. It includes information about frame timing and is used as a heartbeat. It can give you a quick idea in the monitor panel about all your providers and how they are going. That setting controls the rate at which UE sends these messages.

Enable Hitch Detection

Enable basic hitch detection logic. This will use the stat thread so messages will be drawn on the viewport, including render nodes. To avoid that, you need to launch with -ExecCmds="DisableAllScreenMessages".

Minimum Frame Rate

When rendering or game thread takes more than that frame rate interval, a hitch event will be triggered.

Export Settings

Stage_PluginExportSettings.png

Setting Name

Description

Keep Only Last Period Message

When exporting to JSON, only export the last message for all period types of events. This is used to reduce file size.

Exclude Message Types

Add the message type you want to exclude when exporting to JSON.

언리얼 엔진 문서의 미래를 함께 만들어주세요! 더 나은 서비스를 제공할 수 있도록 문서 사용에 대한 피드백을 주세요.
설문조사에 참여해 주세요
건너뛰기