Unreal Engine 5 Documentation has been moved to the Epic Developer Community

Aligning the LED Wall to Camera Tracking using Arucos

A guide to aligning an optical tracking system to an LED wall using Aruco markers displayed on the LED wall.

Choose your operating system:

Windows

macOS

Linux

Goals

In this Quick Start guide, you will go through the steps to align an LED wall to your camera tracking with Aruco markers.

Objectives

  • Generate Aruco markers for your LED wall.

  • Preview the Aruco markers in the editor Viewport and the stage LED screens.

  • Use the Aruco markers to calibrate your LED wall.

1 - Required Setup

For this guide, you will use a production camera, an optical camera tracking system, and an AJA Kona 5 for source video input. In addition, you will need an existing LED wall configuration and a nDisplay Blueprint that matches your stage LED wall configuration. Follow the Rendering to Multiple Displays with nDisplay documentation to learn how to create your nDisplay Blueprint.

Before proceeding, follow the Camera Calibration for Production guide to make sure the lens distortion and nodal point offset of your camera are calibrated. This will create a Lens File Asset that will be used as part of the LED wall alignment with camera tracking.

The example in this guide is based on the Epic LA RnD stage LED wall setup.

The Epic LA RnD stage

Click image to expand.

2 - Generating the Aruco markers

  1. Click Settings > Plugins to open the Plugins Menu .

    Open the Plugins Menu

  2. Search for the LED Wall Calibration and the Live Link Over nDisplay plugins and enable them. Restart the editor.

    Enable the LED Wall Calibration plugin

    Enable the Live Link Over nDisplay plugin

  3. In the Content Browser , search for your nDisplay stage actor Blueprint and double-click it to open it.

    Double-click the nDisplay stage actor Blueprint

    Open the nDisplay Blueprint

    Click image to expand.

  4. Select your first wall mesh from the Components panel and double-click the corresponding cluster from the Cluster panel. This will give you the selected display's resolution.

    Double-click the corresponding cluster from the Cluster** **panel

  5. With the wall mesh selected, click + Add Component and search for and select Calibration Point . This will add a Calibration Point component as a child of your wall mesh.

    Click + Add Component and search for and select Calibration Point

  6. Rename the Calibration point to ArucoW1 .

    Rename the Calibration point to ArucoW1

  7. With the Calibration Point selected, go to the Details panel and scroll down to the Calibration section. Click Create Arucos .

    Add a Calibration Point component

  8. The Aruco Generation Options window will open.

    1. Enter the Texture Width and Texture Height values that correspond to the wall viewport that Arucos are being applied to.

    2. Select the appropriate Aruco Dictionary from the list. In this example, the DICT_6x6_1000 dictionary is selected, which has 1000 unique Aruco symbols.

    3. Enter the Marker ID that will be used as the starting ID for the Aruco markers that will be generated.

    4. Enter the Place Modulus value. This value represents the number of LED panels that will be skipped when placing the Aruco markers. This is useful for working on a large LED wall, where you would use over 1000 unique Aruco symbols if they were to be placed on every LED panel. The default value is 2. Change this to 1 for smaller LED walls where one Aruco can be displayed on every panel. Using the default value of 2 will place an Aruco symbol on every second LED panel.

    5. Click OK to generate the Aruco markers.

      Enter the Aruco generation options

  9. Select a location to save the texture and click OK . A confirmation window will appear that tells you which marker Id to start from for the next viewport. This will be remembered by UE, but can be adjusted in case you need to change the order. Click OK to confirm.

    Select a save destination for the texture

    Confirmation message after saving the texture

  10. Select the Viewport to which the calibration point was applied in the Cluster tab.

    Select the Viewport where the calibration point was applied to in the Cluster tab

  11. Go to the Details panel on the right. Scroll down to the Texture Replacement section and select the generated texture as the Source Texture . Enable the Enable Viewport Texture Replacement checkbox.

    Add the generated texture as the Source Texture

  12. Repeat steps 5 to 10 to add the Aruco markers to the remaining LED viewports.

  13. In the Content Browser , double-click the nDisplay Blueprint to open it.

    Double-click the nDisplay stage actor Blueprint

  14. Click + Variable to create a new boolean variable and name it Replace_Viewport_Textures . Go to the Details panel and enable the Instance Editable checkbox.

    Create a new variable and call it Replace_Viewport_Textures

    Enable the Instance Editable checkbox

  15. Drag Replace_Viewport_Textures to the Event Graph and select Get Replace_Viewport_Textures to create a node.

  16. Right-click in the Event Graph and search for and select Set Replace Texture Flag for All Viewports .

  17. Connect the Event Tick node to the Set Replace Texture Flag for All Viewports node. Connect the Replace Viewport Textures node to the Replace pin of the Set Replace Texture Flag for All Viewports node.

    Connect the Event Tick node to the Set Replace Texture Flag for All Viewports node

  18. Compile and Save the Blueprint.

  19. To see the changes in the Preview Pane of the nDisplay Blueprint, select the nDisplay Blueprint and go to the Details panel. Scroll down to the Editor Preview section and enable the Replace Viewport Textures and Enable Editor Preview checkboxes. This will show the Aruco markers.

    Select the nDisplay Blueprint

    Enable the Replace Viewport Textures and Enable Editor Preview checkboxes

    You can see the Aruco markers on all display viewports

    Click image to expand.

  20. If you are using multiple render nodes, sync the project to the other computers now.

  21. In the World Outliner , select the nDisplay Blueprint and go to the Details panel. Scroll down to the Default tab and enable the Replace Viewport Textures checkbox. The Aruco markers will now appear on the LED wall.

    Enable the Replace Viewport Textures checkbox

    The Aruco markers now appear on the LED wall

    Click image to expand.

  22. To see the Aruco markers in the Editor Viewport, scroll down to the Editor Preview section and enable the Enable Editor Preview checkbox.

    Enable Editor Preview checkbox

    You can now see the Aruco markers in the Editor Viewport

    Click image to expand.

  23. To check the alignment of the LED wall, it is helpful to see the points at the corners of the Aruco markers.

    1. Select a Calibration Point that was created when the Arucos were added.

    2. Go to the Details panel and scroll down to the Calibration section. Enable the Visualize Points in Editor checkbox.

    3. The Point Visualization Scale adjusts the scale of the markers on the mesh. The Visualization Shape dropdown changes the marker shapes between Cubes and Pyramids . Adjust these values to suit your needs.

      Enable the Visualize Points in Editor checkbox

  24. Compile and Save the Blueprint.

    You can now see the corner markers for each Aruco Marker

  25. In the World Outliner window, select the nDisplay Blueprint and go to the Details panel. Scroll down to the Editor Preview section and enable the Enable Editor Preview checkbox.

    This will allow the CG Arucos to be displayed on the viewport mesh in the Lens file. Once the LED calibration is applied, the CG Arucos will be overlaid on the live video feed from the camera that shows the Arucos on the LED wall.

    Enable Editor Preview checkbox

  26. Place the tracked production camera and any other tracked objects under a Tracking Node . Parent the Tracking Node to the nDisplay Blueprint . In this example, a TrackingOrigin has been parented below the NDC_LARD nDisplay Blueprint . After calibrating the LED wall, the offset calculated is applied to the TrackingOrigin .

    Place the tracked production camera and any other tracked objects under a Tracking Node

Section Results

You generated the Aruco markers and are now ready to use them as part of your Lens file to calibrate your LED wall.

3 - Calibrating your LED wall with the Aruco markers

  1. In the Content Browser , double-click the Lens File that was created in the Camera Calibration for Production guide to open it. The Lens File should be attached to the camera under the LiveLinkController streaming the FIZ data.

    1. Click the Nodal Offset tab.

      Open the Lens File and click the Nodal Offset tab

      Click image to expand.

    2. Go to the Viewport Settings and set the Transparency to 0 . This will turn off the CG overlay of the Aruco markers that are displayed in the Editor.

      Set the Transparency to 0

    3. Go to the Nodal Offset section and set the Nodal Offset Algo to Nodal Offset Aruco markers

    4. Set the Calibrator to the nDisplay Blueprint . In this example it is called NDC_LARD_2 .

    5. Enable the Show Detection checkbox . This will capture an image each time the Viewport is clicked and will be used to gather calibration points of the Aruco Markers.

      Adjust the Nodal Offset settings

  2. Point the production camera at the LED wall that is displaying the Arucos markers. You will now collect images of the Aruco markers across the LED wall. Click the mouse inside the viewport to capture an image. A preview of the captured image will be displayed. Click the OK button.

    A preview of the captured image will be displayed

    Click image to expand.

  3. Repeat the previous step multiple times to capture several images across the LED wall. You will see the captured images displayed under the Nodal Offset section.

    The captured images will be displayed on a list

  4. Once you have captured enough samples across the LED wall, click on Apply to Camera Parent button. This will offset the Tracking Origin previously created and the tracked production camera below it.

    The Location and Rotation values are now adjusted with the correct offset values

  5. To check the alignment of the LED wall, set the Transparency to 0.5. This will show the CG Arucos applied to the LED wall mesh overlaying the Arucos on the physical LED wall. Both sets of Aruco markers should now be right on top of each other. If not, the virtual LED mesh does not match the physical LED mesh.

    Check the alignment of the Aruco markers

Section Results

You calibrated your stage LED wall using the Aruco markers.

Help shape the future of Unreal Engine documentation! Tell us how we're doing so we can serve you better.
Take our survey
Dismiss