Interacting with the Pixel Streaming System

Ways your Unreal Engine application can interact with the Pixel Streaming system at runtime.

Choose your operating system:

Windows

macOS

Linux

When you have the Pixel Streaming system set up as described in the Quick Start guide you don't need to do anything special in the gameplay code of your Unreal Engine application to manage the system at runtime. However, the Pixel Streaming Plugin does offer some optional ways for you control and interact with the Pixel Streaming system to achieve specific effects. This page describes these additional options.

Freeze Frame

Encoding every frame rendered by the Unreal Engine application into a media stream has some downsides: it takes resources on the computer running Unreal Engine, sending all of those frames over the network uses up bandwidth, and depending on the available bandwidth the encoding may degrade the quality of the rendered images.

To minimize the use of GPU resources and network bandwidth, you may at times want to temporarily disable the Pixel Streaming Plugin from encoding and sending every frame, showing connected clients a single static image instead. For example:

  • If the client is not actively interacting with the application and nothing in the virtual world is moving, you may want your application to freeze on the last rendered frame until something changes.

  • You may want to show clients an arbitrary image, such as a loading screen or a message caption.

To allow this, the Pixel Streaming Plugin offers Blueprint nodes that you can use to pause and resume encoding. You'll find these nodes in the Pixel Streaming Freeze Frame category in the Blueprint Editor:

Pixel Streaming Freeze Frame functions

  • When you want to replace the media stream with a static image, use the Freeze Frame node.

    Freeze Frame node

    This node accepts an optional reference to a Texture Asset. If you provide one, connected clients will see the Texture you specify in their player window. If you don't provide one, connected clients will see the last rendered frame generated by the Unreal Engine application at the time you called this node.

  • When you want to resume streaming every frame, use the Unfreeze Frame node.

    Unfreeze Frame node

Freezing on a single frame or image texture does not affect input coming from the browser. By default, the player page will still forward keyboard and mouse events to the Unreal Engine while frozen.

Communicating from UE5 to the Player Page

You can make your Unreal Engine application emit custom events to all connected player HTML pages, which you can respond to in the player's JavaScript environment. This lets you change your web page UI in response to gameplay events.

To set this up:

  1. In your Unreal Engine application, any time you want to emit an event to the player page, use the Pixel Streaming > Send Pixel Streaming Response node. Specify a custom string argument to the node to indicate to the player page what event has happened.

    Click to enlarge image.

Or for a more extensive example (this one sends a response to the player when they hit the 3 key):

Click to enlarge image.

  1. In the JavaScript of your player page, you'll need to write a custom event handler function that will be invoked each time the page receives a response event from the Unreal Engine application. It will be passed the original string argument that was sent by the Send Pixel Streaming Response node. For example:

    function myHandleResponseFunction(data) {
        console.warn("Response received!");
        switch (data) {
            case "MyCustomEvent":
                ... // handle one type of event
            case "AnotherEvent":
                ... // handle another event
        }
    }
  2. Register your listener function by calling the addResponseEventListener function provided by app.js. You pass this function a unique name for your event listener, and your function. For example:

    addResponseEventListener("handle_responses", myHandleResponseFunction);
  3. If you ever need to remove your event listener, call removeResponseEventListener and pass the same name. For example:

    removeResponseEventListener("handle_responses");

If you want to pass more complex data, you can format the string you pass to the Send Pixel Streaming Response node as JSON. For example:
Send Pixel Streaming response using JSON

Responding to Pixel Streaming Events

The Pixel Streaming system offers a way for your application's gameplay code to respond to selected events that occur over the lifetime of the Pixel Streaming session. These events come in two flavors: blueprint events and native C++ events:

Blueprint Events

These delegates can be accessed through the UE5's Blueprint Editor.

Event

Description

OnConnectedToSignallingServer

Emitted when this instance of the Unreal Engine application connects to the Cirrus Signalling Server.

OnDisconnectedFromSignallingServer

Emitted when this instance of the Unreal Engine application disconnects from the Cirrus Signalling Server.

OnNewConnection

Emitted when a new client connects to the instance of the Unreal Engine application.

OnConnectionClosed

Emitted when a client that is connected to this instance of the Unreal Engine application disconnects.

On All Connections Closed

Emitted when the last client that is connected to this instance of the Unreal Engine application disconnects. After this event is emitted, there will be no clients viewing your application's media stream. You might use this as an opportunity for your gameplay logic to reset the application to an initial state in preparation for a new client to join.

OnStatChanged

Reports different pixel streaming stats in the format of FPixelStreamingPlayerId, FName, float (Player ID, stat name and stat value).

To respond to these events, you bind to them using the Pixel Streamer Delegates class. You'll typically set up this binding at the beginning of gameplay; for example, you might do this in response to the Event BeginPlay event. Once the binding is set up, your custom event will get triggered each time the bound event is triggered.

  1. In the Blueprint Graph Editor, drag right from any node's execution pin and choose Pixel Streamer Delegates > Get Pixel Streamer Delegates.

    Get Pixel Streamer Delegates

  2. Drag right from its Return Value and expand the Pixel Streamer Delegates category. You'll see a number of new options for binding and unbinding events.

    Pixel Streamer Delegates nodes

  3. Choose the Bind... option for the event you want to respond to, such as Bind Event to On All Connections Closed. You'll get a new node with an Event input. Connect its input execution pin to the output execution pin from Get Pixel Streamer Delegates.

    Bind Delegate

  4. If you already have a custom event that you want to trigger, connect the Output Delegate pin in its title bar to the Event input on the Bind node you just created. Otherwise, drag left from the Event input and choose Add Event > Add Custom Event to create a new custom event.

    Custom Event

  5. Connect your custom event to the Blueprint logic you want to run in response to the Pixel Streaming event.

For example, this implementation calls a custom function defined in the same Blueprint to reset the application to its initial state:

Respond to the custom event

C++ Events

Developers can register their own handlers to these Pixel Streaming delegates. We recommend developers refer to this page https://docs.unrealengine.com/5.0/en-US/multicast-delegates-in-unreal-engine for the specific details on C++ delegate usage in Unreal Engine. A simple example of using the Pixel Streaming delegates is:

#include "PixelStreamingPlayerId.h"
#include "PixelStreamingDelegates.h"

void FExample::OnStatChanged(FPixelStreamingPlayerId PlayerId, FName StatName, float StatValue)
{
//TODO: Do something with the stat that changed.
}

void FExample::BindToDelegates()
{
// Bind to the OnStatChangedNative delegate.
if(UPixelStreamingDelegates* Delegates = UPixelStreamingDelegates::GetPixelStreamingDelegates())
{
Delegates->OnStatChangedNative.AddRaw(this, &FExample::OnStatChanged);
}
};