Temporal Super Resolution

A high-level overview of the Anti Aliasing options available in Unreal Engine.

Temporal Super Resolution (TSR) is a platform-agnostic Temporal Upscaler that enables Unreal Engine to render beautiful 4K images. Images come at a fraction of the cost by amortizing some of the costly rendering calculations across many frames. It does this by rendering lower internal resolution than what Temporal Anti-Aliasing Upsampling (TAAU) in Unreal Engine 4 could do.

TSR provides a native high-quality upsampling technique to meet the demand of next-generation games. It realizes the possibilities needed with the fidelity and details required by Nanite geometry while allowing rendering frames at much lower resolution to accommodate enough performance for Lumen.

The comparison below demonstrates the quality and performance difference between frames rendered at native 4K and those rendered at 1080p upscaled to 4K. With TSR, it's possible to achieve image quality near 4K resolution while also reducing GPU frame time by half.

4k frames rendered at native 4K resolution | Frame time: 57.50ms

4k frames rendered at 1080p resolution (r.ScreenPercentage=50) | Frame Time 33.37ms

Each of the images in the comparison above are 4K images that are limited to the width of this page. To see their fully uncompressed resolution, right-click on either and save them to your computer, or open them in a new browser window.

Temporal Super Resolution has the following properties:

  • Rendered frames approach the quality of native 4K with input resolutions as low as 1080p.

  • Less "ghosting" artifacts against high-frequency backgrounds than was visible with Unreal Engine 4's default Temporal Anti-Aliasing method.

  • Reduced flickering on geometry with high complexity.

  • Supports Dynamic Resolution scaling on console platforms.

  • Runs on any hardware that supports D3D11, D3D12, Vulkan, Metal, PlayStation 5, and Xbox Series S | X.

  • Shaders are optimized specifically for PlayStation 5 and Xbox Series S | X GPU architectures.

In the rendering chain, TSR happens after the depth of field and everything that follows is upscaled.

8-pipeline-tsr.png

TSR's Scalability

You can use Temporal Super Resolution across supported platforms, but you will want to customize the upscaling settings of each platform based on the needs of your project. Explore the sections below to learn some of the ways you can go about checking TSR in your project and then scaling it appropriately.

Understanding the Caveats of Temporal Accumulation of Details

Advertising only the increases to frame rate that TSR allows by rendering at lower resolutions wouldn't entirely be fair since temporal upscalers in general have their own caveats. Given enough frame time with a static image, any temporal upscaler could render identical results. Also like any temporal upscaler, TSR accumulates lower resolution frames over time to converge an image, and the detail of an image can only be known after enough details are accumulated. For instance, how thick any geometry is cannot be known in the first frame.

r.AntiAliasingMethod 4 | r.Test.SecondaryUpscaleOverride 4 | ScreenPercentage=61

r.AntiAliasingMethod 0 | r.Test.SecondaryUpscaleOverride 4 | ScreenPercentage=61

The rendering resolution is controlled by the Screen Percentage. This controls how much information is available for one frame. The rest of the information needed to converge depends on the remaining parts of the frame to be rendered. This means that information displayed by TSR is dependent on both the resolution of the frame being rendered and the frame rate because it influences how fast details can accumulate.

More than just the GPU can limit frame rate that affects TSR, like when CPU bound or VSync on non-variable refresh rate monitors.

This is where the console command stat tsr can help identify what may affect TSR's accumulation to the overall image. When called, the command adds two stats to the ones normally displayed when calling stat unit. These are TSR feed and TSR1spp.

Display stats when calling "stat tsr."

TSR feed shows the number of pixels per second being fed into TSR, which is an important macroscopic metric to understand how much data TSR has to converge to the overall image. Rendering Resolution Width * Rendering Resolution Height * FrameRate = Display Resolution Width * Display Resolution Height * ScreenPercentage^2 * FrameRate. The benefit of this metric is it indicates the macroscopic image quality in motion regardless of the display resolution. To illustrate how this scales:

Display Resolution

Screen Percentage

Framerate

TSR Feed in MP/s

4k (3840x2160)

50%

60hz

3840*2160*(50/100)^2*60 = 124.4 MP/s

4k (3840x2160)

58%

60hz

167.4 MP/s

4k (3840x2160)

66%

60hz

216.7 MP/s

4k (3840x2160)

50%

30hz

62.2 MP/s

4k (3840x2160)

72% = 50% x sqrt(2)

30hz

124.4 MP/s

1080p (1920x1080)

100%

60hz

124.4 MP/s

1080p (1920x1080)

72% = sqrt(0.5)

60hz

62.2 MP/s

1080p (1920x1080)

50%

60hz

31.1 MP/s

TSR 1spp is TSR's convergence rate — the time needed for TSR to have enough data to reach one sample per pixel. It is especially important in motion where there can be disocclusion that needs to accumulate details quickly. TSR 1spp = 1000 / (ScreenPercentage^2 * FrameRate).

Screen Percentage

Framerate

TSR Convergence Rate

50%

60hz

1000 / ((50/100)^2 * 60) = 66.6 ms

58%

60hz

49.5 ms

66%

60hz

38.2 ms

100%

60hz

16.6 ms

50%

30hz

133.3 ms

An example situation of how to use this data would be if your Screen Percentage were set to 50, this means you need (50/100)^-2 = 4 frames to have at least one sample per pixel. If each frame takes 16.6 milliseconds to render, it means a disocclusion area on screen will take four times that to have enough data or 4*16.6 = 66.4ms.

TSR's Upscaling GPU Cost

The primary goal of TSR is upscaling. The majority of its GPU work scales based on the resolution it is given. That is due to some of the GPU cost of TSR needing to be done at a higher display resolution than the rendering resolution.

tsr-gpucost-1.png

tsr-gpucost-2.png

Open the GPU statistical display from the console using stat gpu. With this open, you can adjust the primary Screen Percentage to see the difference in performance when rendering at 100 percent versus 50% screen percentage, like the examples taken below from the Valley of the Ancients sample project.

~0.79ms at r.ScreenPercentage=100

~0.43ms at r.ScreenPercentage=50

Click image for full size.

Click image for full size.

The parallax heuristic of TSR depends on the depth and velocity buffer rather than the scene color and translucencies of the frame because the depth and velocity buffers often finish on the GPU much earlier than the scene color and translucency buffers do. This allows the entire TSR parallax heuristic to compute asynchronously on the GPU, filling in the gaps where the GPU is underused by other rendering algorithms with r.TSR.AsyncCompute=2.

In Fortnite Chapter 4 on PlayStation 5 and Xbox Series X, this is offsetting approximately 0.5ms of TSR's total GPU cost when testing performance over an entire Battle Royal performance replay. In the replay, about 0.1ms is saved, bringing down the effective TSR cost to 1.5ms and critical path GPU cost to finish rendering the frame to 1.1ms.

tsr-gpucost-3.png

Console Variables

The Engine Scalability Settings scale the quality of TSR across all platforms. They use predefined scalability groups from Low to Cinematic. Each group is defined by a set of console variables that affect different rendering properties, including anti-aliasing.

You can inspect how the scalability settings are used by opening the BaseScalability.ini file in the [Unreal Engine Root]/Engine/Config folder. Look at the AntiAliasingQuality sections to see how TSR is scaled depending on the anti-aliasing quality group being used. You can modify the anti-aliasing quality groups for your own project by changing the settings in [Your Project Root]\Config\DefaultScalability.ini.

For example, an adjustment you may want to make for your project is that TSR includes a spatial anti-aliasing algorithm that is used whenever the history needs to be rejected to avoid stale data that could introduce ghosting errors to the current frame. Game projects that still want to render at a higher resolution, or higher frame rate, may not need this option since the visible aliasing may not be subject to rejection. History rejection for TSR is controlled with r.TSR.RejectionAntiAliasingQuality.

Hidden Additional GPU Costs of Temporal Upscaling

Temporal Super Resolution and any other temporal upscaler happens in the middle of the post-processing chain of the renderer. This means that passes, such as Motion Blur or Tonemapper, running after TSR are no longer scaling with the primary Screen Percentage. In the absence of a secondary screen percentage with a spatial upscaler (r.SecondaryScreenPercentage.GameViewport), the output resolution of TSR runs exactly at the display resolution rather than the rendering resolution. With TSR enabled by default, a lot of effort is focused on reducing the hidden temporal upscaling costs of passes that happen after TSR.

Motion Blur

Motion Blur optimizations combine for 3x motion blur GPU performance cost improvements on PlayStation 5 and Xbox Series X, platforms that always display 2160p backbuffer, even on lower resolution televisions.

  • Increased TSR costs when using stat gpu:

    • TSR cost may increase slightly when using stat gpu when comparing Motion Blur enabled and disabled.

    • When Motion Blur is enabled, TSR takes over its Velocity Flatten pass that normally runs at input resolution. This is controlled using the console command r.MotionBlur.AllowExternalVelocityFlatten, and it is enabled by default.

    • TSR outputs half-resolution Scene Color to reduce bottlenecks in memory bandwidth that can occur in large directional blur kernels. This is controlled using the console command r.MotionBlur.HalfResInput, and it is enabled by default.

  • Half-resolution on directional blur:

    • Directional blur happening at display resolution is optimized to automatically run at half-resolution on very large movements. Enabling this feature will reduce VALU cost of motion blur.

    • Enable this optimization using the console command r.MotionBlur.HalfResGather 1.

  • Motion Blur Half and Quarter Resolution:

    • TSR and Motion Blur are able to output half or quarter resolution, This is of importance for Gaussian and Convolution Bloom, Lens Flare, Eye Adaptation (Auto Exposure), and Local Exposure that all run after Motion Blur.

The PostProcessQuality scalability group scales some of these r.MotionBlur.* in Engine/Config/BaseScalability.ini.

Additional Notes

Troubleshooting TSR

With TSR being a temporal upscaler, there are times you'll need to diagnose artifacts that affect temporal upscalers. You can use the VisualizeTemporalUpscaler showflag to look at the raw input and output buffers as a starting point.

You can access this show flag from the Level Editor's Show > Visualize menu by selecting Temporal Upscaler (TSR, TAAU, or third party plugins).

vis-temporalupscaler.png

For more information, see Temporal Upscalers.

Adjusting for Pixel Flickering

The flickering of pixels in the frame can happen for various reasons and sometimes can be mitigated. There are console variables to define the difference between pixel flickering, like when looking at a building at a grazing angle, or in a pixel shader animation.

You can use r.TSR.ShadingRejection.Flickering.Period to set the period in 60hz frames in which luma oscillations at equal or greater frequency are considered flickering and should be stabilized. But this stabilization can introduce ghosting when it happens on undesired areas of the image

tsr-flickering-2.gif

tsr-flickering-1.gif

r.TSR.ShadingRejection.Flickering.Period: 0

r.TSR.ShadingRejection.Flickering.Period: 3 (Default)

Screen Percentage: 100

Screen Percentage: 100

This setting of TSR is important because it is what differentiates a pixel flickering from other desired visual effects, like pixel shader animation. However, an important difference is that flickering happens regardless of frame rate. Visual effects that are based on time and are therefore independent of the frame rate. This can mean that a visual effect that looks smooth at 60hz might appear to "flicker" at lower frame rates, like 24hz.

It's important that visuals authored by an artist remain unaffected by frame rate since TSR scales to meet GPU performance. This can have the unintended effect of altering visuals when a player's machine is running at lower than expected frame rates. For this purpose, the console command r.TSR.ShadingRejection.Flickering.Period automatically reduces anti-flickering when the frame rate drops lower than the frame rate defined with r.TSR.ShadingRejection.Flickering.FrameRateCap (default is 60hz). This means that geometric details that might be stable at 60hz might become less stable at a lower frame rate. This behavior is enabled by default but can be opted out of with r.TSR.ShadingRejection.Flickering.AdjustToFrameRate.

Supported Platforms

Temporal Super Resolution is available on desktop renderers across all desktop hardware that supports Shader Model 5. Temporal Super Resolution is supported on the following platforms:

  • Windows D3D11 SM5, D3D12 SM5, D3D12 SM6, and Vulkan SM5

  • Linux Vulkan SM5

  • Mac Metal SM5

  • PlayStation 5 and Xbox Series S | X

TSR's upscaling quality and behavior is strictly identical across all supported platforms. However, TSR has been specifically optimized for AMD RDNA GPUs used in PlayStation 5 and Xbox Series S | X consoles, taking advantage of 16-bit types and packed instructions.

Console Variables

Console Variable Name

Description

r.TSR.AsyncCompute

Controls how TSR runs on async compute. Some TSR passes can overlap with previous passes:

  • 0 is disabled (default).

  • 1 runs on async compute-only passes that are completely independent from any intermediary resource of this frame, namely ClearPrevTextures and ForwardScatterDepth passes.

  • 2 runs on async compute-only passes that are completely independent or only dependent on the depth and velocity buffer, which can overlap, such as translucency and depth of field. Any passes on the critical path remain in the graphics queue.

  • 3 runs all passes on async compute.

r.TSR.History.R11G11B10

Select the bit depth of the history when set to "1". It saves memory bandwidth that is of particular interest in TSR's UpdateHistory's runtime performance by saving memory both at the previous frame's history reprojection and its write-out of the output and new history. This optimization is unsupported when r.PostProcessing.PropagateAlpha=1 is used. Also, note that increasing r.TSR.History.ScreenPercentage to 200 adds two additional implicit encoding bits in the history compared to the TSR Output's bit depth because of a downscaling pass from TSR history resolution to TSR output resolution.

r.TSR.History.SampleCount

The maximum number of samples for each output pixel in the TSR history. Higher values means more stability on highlights on static images but it may introduce additional ghosting on some styles of VFX, such as fireflies. You can have a minimum of 8 samples and a maximum of 32 due to the encoding of TSR.History.Metadata. The default sample count is 16.

r.TSR.History.ScreenPercentage

The resolution multiplier of the history of TSR based off the output resolution. Increasing the resolution adds runtime cost to TSR but allows it to maintain a better sharpness and stability with the details stored in history throughout the reprojection. By default, the value is set to 200 because a particular property relying on the NyQuist-Shannon sampling theorem is used that establishes a sufficient condition for the sample rate of the accumulated details in the history. As a result, only values between 100 and 200 are supported. This value is controlled with the Anti-Aliasing scalability group. Epic and Cinematic quality levels use 200, while all others use 100.

r.TSR.History.UpdateQuality

Selects shader permutation of the quality of the update of the history in the TSRHistoryUpdate pass. This is currently driven by the sg.AntiAliasingQuality scalability group. For details about what each group offers, see DIM_UPDATE_QUALITY in TSRUpdateHistory.usf to make any customizations.

r.TSR.RejectionAntiAliasingQuality

Controls the quality of TSR's built-in spatial anti-aliasing technology when the history needs to be rejected. While this is not critical when the rendering resolution is not much lower than the display resolution, this technique is essential to hiding lower rendering resolution because of two reasons: the screen space size of aliasing is inverse proportional to the rendering resolution, and rendering at a lower resolution means needing more frames to reach at least 1 rendered pixel per display. By default, this option is enabled for all Anti-Aliasing scalability groups except the Low scalability group.

r.TSR.ShadingRejection.Flickering

Instability in TSR's output comes from instability of the shading rejection the majority of the time. This can happen for different reasons, such as:

  • The most common source of instability is the Moire pattern between structured geometry and the rendering pixel grid changing every frame due to the offset of the jittering pixel grid offset.

  • Another source is with extreme geometric complexity due to temporal history's chick-and-egg problem that cannot be overcome by other mechanisms in place in TSR's RejectionHistory pass — like, how can the history be identical to a rendered frame if the amount of details available in the frame is not already in the history? Or, how can the history accumulate details if the history is too different from the rendered frame?

When enabled, this heuristic monitors the luminance of the frame right before any translucency drawing stored in TSR.Moire.Luma resource evolves over successive frames. If flickering is constantly detected regularly happening above the threshold defined by r.TSR.ShadingRejection.Flickering console variable, the heuristic attempts to stabilize the image by letting ghosting happen within the luminance boundary tied to the amplitude of flickering. A caveat of this heuristic is that any opaque geometry with an incorrect motion vector can make a pixel look identical to flickering causing this heuristic to initiate and leaving undesired ghosting effects on affected geometry. When this happens, you should verify how the motion vector looks through the VisualizeMotionBlur show flag and how these motion vectors are able to reproject the previous frame with VisualizeReprojection show flag. Use the console variable r.TSR.ShadingRejection.Flickering.Period to control the frame frequency at which a pixel is considered to be flickering and needs stabilizing. For instance, if the flickering period is set to 3, any pixel that changes luminance equal to this or more per frame is considered flickering.

Another caveat of determining what is considered a flickering pixel versus an animated pixel is that flickering happens regardless of frame rate, whereas visual effects that are/should be based on time and are therefore independent of the frame rate. This can mean that a visual effect that looks smooth at 60 frames per second might appear to "flicker" at lower frame rates, like 24 fps. To avoid ghosting with artist-authored visual effects, the console variable r.TSR.ShadingRejection.Flickering.AdjustToFrameRate (enabled by default) looks for changes that drop below the one set by r.TSR.ShadingRejection.Flickering.FrameRateCap. While r.TSR.ShadingRejection.Flickering is controlled based on scalability settings to turn this heuristic on or off depending on lower or higher end GPUs. You could also set this console variable in your project's DefaultEngine.ini for consistency across all platforms.

By default, this console variable is enabled in the Anti-Aliasing scalability group for High, Epic, and Cinematic levels.

r.TSR.ShadingRejection.Flickering.AdjustToFrameRate

Whether the flickering period settings (r.TSR.ShadingRejection.Flickering.Period) should adjust to the frame rate when lower than this frame rate cap. see r.TSR.ShadingRejection.Flickering for more details.

r.TSR.ShadingRejection.Flickering.FrameRateCap

Framerate cap in hertz at which there is automatic adjustment of r.TSR.ShadingRejection.Flickering.Period when the rendering frame rate is lower. See the r.TSR.ShadingRejection.Flickering entry for more details. By default, this is set to 60hz.

r.TSR.ShadingRejection.Flickering.MaxParallaxVelocity

Some materials that use Parallax Occlusion Mapping, such as City Sample with its buildings' windows interiors can often not accurately render a motion vector of this faked interior geometry. This can cause the heuristic to believe it is flickering, when it's not. This variable defines the parallax velocity in 1080p pixel at the frame rate defined by r.TSR.ShadingRejection.Flickering.FrameRateCap at which point the heuristic should disable to not cause ghosting.

r.TSR.ShadingRejection.Flickering.Period

Period in frames in which luma oscillations at equal or greater frequency is considered flickering and should ghost to stabilize the image. See the r.TSR.ShadingRejection.Flickering entry for more details. By default, this is set to 3.

r.TSR.ShadingRejection.SampleCount

Maximum number of samples in each output pixel of the history after total shading rejection. Lower values means higher clarity of the image after shading rejection of the history but at the trade off of higher instability of the pixel on subsequent frames accumulating new details, which can be visually distracting. By default, this is set to 2.0.

r.TSR.Subpixel.DepthMaxAge

Maximum age (in frames) of subpixel's depth in history for their self reprojection. Default is 3 frames.

r.TSR.Subpixel.IncludeMovingDepth

Whether the depth of moving subpiel detail should also include the subpixel depth history for their reprojection. In most cases this should not be enabled since it's impossible to know how a moving objects vlocity evolves over time when it's only drawing its velocity occasionally. This setting is disabled by default.

r.TSR.Subpixel.Method

A particular challenge with the amount of detail that Nanite can render is that sometimes a mesh's details can be thinner than a rendered pixel, which causes them to only render for some frames. When this happens, it means neither the depth nor velocity buffers are able to reproject them. You can visualize this with the vis SceneDepthZ command.

This setting controls the method to reproject and discard subpixel details:

  • 0 disables subpixel details accumulation in history, which means all subpixel features may have ghosting.

  • 1 accumulates how much subpixel detail can parallax under movement with their background history rejection. It allows a minimal amount of ghosting but at the expense of some image stability on very thin geometry.

  • 2 accumulates the subpixel detail's closest depth to be able to reproject them even when they are now drawing in the depth and velocity buffers. This works well for static geometry but not so much for moving geometry. (This is the default method.)

r.TSR.Velocity.WeightClampingPixelSpeed

Defines the output pixel velocity at which the high frequencies of the history gets their contributing weight clamped. It's basically to lerp the effect of r.TSR.Velocity.WeightClampingSampleCount when the pixel velocity gets smaller than r.TSR.Velocity.WeightClampingPixelSpeed.

r.TSR.Velocity.WeightClampingSampleCount

The number of samples to count to in the history pixel to clamp history when the output velocity is reached by r.TSR.Velocity.WeightClampingPixelSpeed. Higher values means higher stability on movement but at the expense of additional blur due to successive convolution of each history reprojection. You can visualize the number of samples in TSR history with the console command vis TSR.History.Metadata. Note that this clamps the sample count in pixel history, not the output pixel. Therefore, lower values, by design, are less noticeable with higher TSR screen percentage (r.TSR.History.ScreenPercentage). This is done so that increasing it unilaterally and automatically gives more temporal stability while maintaining sharpness of detail reprojection at the expense of extra runtime costs regardless of this setting. For instance, a story-driven game might prefer to keep this setting at 4.0 for a "cinematic" look whereas a competitive game like Fortnite would prefer to lower that to something like 2.0. (Default is 4.0f.)

r.TSR.WaveOps

Whether to use wave ops in the shading rejection heuristics to speed up convolutions. The shading rejection heuristic optimization can be hard for the shader compiler, causing them to become corrupt or show quality loss.

This optimization is currently disabled on SPIR-V platforms (mainly Vulkan and Metal) due to adding to compilation time in the SPIR-V backend with DXC, which can take longer to start the editor.

Help shape the future of Unreal Engine documentation! Tell us how we're doing so we can serve you better.
Take our survey
Cancel