Camera Lens Calibration Quick Start Guide

In this Quick Start guide, you will go through the steps to calibrate lens distortion and the nodal point offset, using the Camera Calibration plugin.

This guide uses a production camera, an optical camera tracking system, and an AJA Kona 5 for source video input in the examples.

Prerequisites

You must set up the following in your project to complete the sections below:

• Camera Calibration plugin enabled.

• Camera tracking and focus, iris, zoom (FIZ) data streaming through Live Link. Specifically, your project must have a CineCamera Actor with a Live Link Component Controller that has a Live Link Virtual Subject with a Camera Role.

• Source video input, such as AJA or Blackmagic.

• A Media Profile added to your project and Media Sources configured.

• (Optional) Timecode and genlock setup in the Media Profile if you're using a sync generator. Refer to Media Profile for how to set this up.

• (Optional) Timed Data Monitor to evaluate sync and timecode.

Step 1 - Creating a Lens File Asset

The Lens File provides the calculation and storage of distortion and nodal offset data, specific to each lens-to-camera combination. We recommend you create a new Lens File if the lens, tracking object location, or the camera body changes. A new Lens File needs to be created for each lens-to-camera body combination.

Follow these steps to create a Lens File Asset in your project:

1. In the Content Browser, right-click to open the context menu and choose Miscellaneous > Lens File to create a new Lens File Asset.

2. Double-click your Lens File Asset to open the Lens File Asset Editor.

3. Under Viewport Settings:

   1. Set Camera to the connected camera.

   2. Set Media Source to your Media Profile.

4. Under Lens Information:

   1. Add a Lens Model Name. A recommended naming convention combines the camera body name and the focal length.

   2. Add the Lens Serial Number if available.

   3. Add the Lens Model. Only spherical lens models are supported at this time.

   4. Set Sensor Dimensions to match your camera.

5. Select your CineCamera Actor in the Outliner to open its Details panel.

   1. In the Details panel, select the Live Link Controller Component to view its details.

   2. Under Role Controllers > CAMERA ROLE > Camera Calibration > Lens File Picker, set Lens File to the Lens File Asset created in the previous section.

At the bottom of the Lens File Asset Editor you will find a summary of the current properties applied to the Lens File. Most of the properties show as blank or N/A at this stage. These properties will update as you continue with the calibration process.

Step 2 - Calculating Lens Distortion with a Checkerboard

This step shows how to calculate lens distortion with the common method of using a checkerboard. You can use a printed checkerboard mounted on a rigid surface and held in view of the production camera, or a checkerboard image displayed on a tablet. You'll create a CG checkerboard in UE to match the properties of the physical checkerboard.

Follow these steps to calculate lens distortion:

1. In the Toolbar, select Add Content and choose Virtual Production > Checkerboard to add a CameraCalibrationCheckerboard Actor to the level.

2. Select the CameraCalibrationCheckerboard Actor in the Outliner to open its Details panel. In the Details panel under Calibration:

   1. Set Number of Corner Rows to the number of corners in one of your checkerboard's columns. In this example, the number of row corners is 7.

   2. Set Number of Corner Columns** to the number of corners in one of your checkerboard's rows. In this example, the number of column corners is 11.

   3. Set Square Side Length to the length of a square in your checkerboard in centimeters (cm). In this example, each square side is 4.5 cm.

   4. Initially there aren't any materials assigned to the checkerboard. Set Odd Cube Material to a Material with the color of the top left square, and Even Cube Material to the color of the square next to it. In this example, the color of the first square is black, and the next square is white.

3. In the Lens File Asset Editor, switch to the Lens Distortion tab.

   1. Set Lens Distortion Algo to Lens Distortion Checkerboard.

   2. Set Checkerboard to the CameraCalibrationCheckerboard Actor you created.

   3. Enable Show Detection. This will create a preview image each time an image is created for use in calculating lens distortion.

4. Place the checkerboard in front of the camera as shown below.

5. Click on the viewport to start the calibration process and create the first image to be used.

6. Move the checkerboard around the camera field of view and continue to click the image to capture more calibration pictures. Make sure to move the checkerboard's location enough times to cover the field of view with overlapping images, at different orientations, and different depths to increase the accuracy of the calibration.

1. After collecting enough overlapping images to at least cover the complete field of view, click Add To Lens Distortion Calibration.

2. The Distortion Parameters at the bottom of the window are updated.

Most lenses will have different distortion values at different focus distances. To make the most accurate Lens File for lens distortion, the process above should be repeated at different focus distances.

Step 3 - Confirming Lens Distortion

To check the lens distortion, you can use the Nodal Offset tab. In this case, the untracked CG checkerboard in your scene overlays the video feed so you can visually confirm whether the physical checkerboard and CG checkerboard line up.

Follow these steps to confirm your lens distortion:

1. Place the physical checkerboard in the field of view of the camera.

2. In the Lens File Asset Editor, switch to the Nodal Offset tab.

   1. Set Nodal Offset Algo to Nodal Offset Checkerboard.

   2. Set Checkerboard to your Checkerboard Actor.

3. Click the image to populate all the corner data from the calibration.

4. Click Apply To Calibrator. This moves the Checkerboard Actor to the camera to match the physical checkerboard.

5. Under Viewport Settings, set Transparency to 1 to verify the Checkerboard Actor matches the physical checkerboard in the camera viewport.

Step 4 - Creating a Tracked Object for Nodal Offset Calibration

The nodal offset accounts for the difference between the axis of the tracked camera and the axis of the actual nodal point of the lens / camera combination being calibrated. If the lens is changed or the tracking device on the camera is moved, a new nodal point offset will need to be calculated.

In order to calculate the nodal offset applied to the CineCamera Actor, you will need to create a physically tracked object and a CG model to match the tracked object.

In this example, the physical checkerboard that was used previously had tracking markers added to it. A CG model was created to match the tracked checkerboard and set up for optical tracking. The CG checkerboard is imported into UE for use in the next steps.

While using a checkerboard is common, you can create any tracked object and CG model to suit your needs. The process will include adding calibration points to your CG model. Your tracked object and CG model must match each other.

Follow the steps below to create a tracked object for nodal offset calibration:

1. In the Content Browser, right-click to open the context menu.

   1. Under Create Basic Asset, choose Blueprint Class.

   2. In the Pick Parent Class panel, select Actor.

2. Double-click the new Blueprint Class in the Content Browser to open it in the Blueprint Editor.

3. In the Components panel, click (+) Add Component and choose Static Mesh.

4. In the Details panel under Static Mesh, set Static Mesh to the CG model of your tracked object. In this example, the CG model is Checkerboard_mesh.

5. In the Components panel, click (+) Add Component and choose Calibration Point. In this example, the component is named Top_Left.

6. In the Viewport, place the Top_Left component on the top left point of the checkerboard, as seen below.

7. Duplicate the Top_Left component four times and name the new copies Bottom_Left, Bottom_Right, Top_Right, and Center.

8. Place those components in their corresponding locations on the checkerboard mesh.

9. Compile and Save the Blueprint.

10. Drag the Blueprint Class into your level.

11. In Live Link, add your tracker source for the checkerboard. In this example, Optitrack is used and the Subject is named Checkerboard. Refer to Live Link for steps on how to add tracker sources.

12. Select your Blueprint Class Actor in the Outliner to open its Details panel. In the Details panel:

    1. Click (+) Add Component and choose Live Link Controller.

    2. Select the Live Link Controller Component to view its details.

    3. Under Live Link, set Subject Representation to your Live Link Subject for your Checkerboard tracker. In this example, the Live Link Subject is Checkerboard.

Step 5 - Calculating Nodal Offset

The nodal offset applies an offset to the CineCamera Actor to account for the actual nodal point of the lens being used on the physical camera.

The Camera Calibration plugin calculates the nodal offset by having the tracked object move to different positions in front of the physical production camera. In this example, the Checkerboard Actor has calibration points which are collected and used to calculate the nodal offset.

Follow these steps to calculate the nodal offset:

1. In the Lens File Asset Editor, switch to the Nodal Offset tab. Under Nodal Offset:

   1. Set Nodal Offset Algo to Nodal Offset Points Method.

   2. Set Calibrator to the Blueprint Class Actor created in the previous section.

2. Click on the checkerboard image in the viewport to match the location of each Calibration Point. For example, click the bottom right corner of the checkerboard to match the Bottom_Right Calibration Point. The Calibration Points update automatically as you click on the image.

3. Move the tracked checkerboard five to eight different positions in front of the camera and acquire a set of calibration points at each position.

4. Click Add To Nodal Offset Calibration.

5. Check that the nodal offset has been applied correctly by checking the Nodal Point Offset properties at the bottom of the editor.

6. Change the transparency from 0 to 0.5. The CG element of the tracked checkerboard should line up with the checkerboard in the live video.

7. Pan the tracked camera and move the tracked checkerboard. The CG version should stay correctly superimposed over the physical checkerboard in the live video feed.

If you did not set up a timecode and genlock, and adjusted synchronization using the Timed Data Monitor, there may be lag in the CG element as you move. When you stop moving the camera, the CG element will line up again.