Implementing ChunkDownloader Ingame

ChunkDownloader is a patching solution for Unreal Engine. It downloads assets from a remote service and mounts them in memory for use in your games, enabling you to provide updates and assets with ease. This guide will show you how to implement ChunkDownloader in your own projects. By the end of this guide you will be able to:

• Enable the ChunkDownloader plugin and add it to your project's dependencies.

• Organize your content into chunks, package them in .pak files, and prepare a manifest file for downloads.

• Implement ChunkDownloader in your game's code to download remote .pak files.

• Access content from mounted .pak files safely.

1. Required Setup and Recommended Assets

Before proceeding any further, you should review the following guides and follow each of their steps:

• Setting Up the ChunkDownloader Plugin

• Preparing Assets for Chunking

• Hosting a Manifest and Assets for ChunkDownloader

These guides will show you how to add the ChunkDownloader plugin to your project, set up a chunking scheme for your assets, and distribute them to a local test server. To review, your example project should be called PatchingDemo, and it should be constructed as follows:

1. It is a C++ project based on a blank template.

2. The ChunkDownloader plugin is enabled in the Plugins menu.

3. Use Pak File and Generate Chunks are both enabled in Project Settings > Project > Packaging.

4. The Boris, Crunch, and Khaimera assets from Paragon are added to the project.

   • You can download these from the Unreal Marketplace for free.

   • You can use any assets you want, as long as they are separated into discrete folders.

5. Each of the three characters' folders has a Primary Asset Label applied to it with the following Chunk IDs:

   Folder	Chunk ID
   ParagonBoris	1001
   ParagonCrunch	1002
   ParagonKhaimera	1003

6. You have cooked your content and have .pak files for each of the above Chunk IDs.

7. There is a manifest file called BuildManifest-Windows.txt containing the following information:

   BuildManifest-Windows.txt

   $NUM_ENTRIES = 3
   $BUILD_ID = PatchingDemoKey
   pakchunk1001-WindowsNoEditor.pak    922604157   ver 1001    /Windows/pakchunk1001-WindowsNoEditor.pak
   pakchunk1002-WindowsNoEditor.pak    2024330549  ver 1002    /Windows/pakchunk1002-WindowsNoEditor.pak
   pakchunk1003-WindowsNoEditor.pak    1973336776  ver 1003    /Windows/pakchunk1003-WindowsNoEditor.pak

   All of the fields for each chunk must be contained on the same line, and they must be separated by tabs, or they will not be parsed correctly.

8. The .pak files and the manifest file are distributed to a locally hosted web site. Refer to Hosting a Manifest and Assets for ChunkDownloader for instrcutions on how to set this up.

9. The DefaultGame.ini file for your project has the CDN URL defined as follows:

   DefaultGame.ini

   [/Script/Plugins.ChunkDownloader PatchingDemoLive]
   +CdnBaseUrls=127.0.0.1/PatchingDemoCDN

2. Initializing and Shutting Down ChunkDownloader

ChunkDownloader is an implementation of the FPlatformChunkInstall interface, one of many interfaces that can interchangeably load different modules depending on what platform your game is running on. All modules need to be loaded and initialized before they can be used, and they also need to be shut down and cleaned up.

The simplest way to do this with ChunkDownloader is through a custom GameInstance class. Not only does GameInstance have appropriate initialization and shutdown functions you can tie into, it also will provide continuous access to ChunkDownloader while your game is running. The following steps will walk you through this implementation.

1. Create a New C++ Class using GameInstance as the base class. Name it PatchingDemoGameInstance.

   Click image to enlarge.

2. Open PatchingDemoGameInstance.h in your IDE. Under a public header, add the following function overrides:

   PatchingDemoGameInstance.h

   public:
   /** Overrides */
       virtual void Init() override;
       virtual void Shutdown() override;

   The Init function runs when your game starts up, making it an ideal place to initialize ChunkDownloader. Similarly, the Shutdown function runs when your game stops, so you can use it to shut down the ChunkDownloader module.

3. In PatchingDemoGameInstance.h, add the following variable declaration under a protected header:

   PatchingDemoGameInstance.h

   protected:
   //Tracks Whether or not our local manifest file is up to date with the one hosted on our website
   bool bIsDownloadManifestUpToDate;

4. Open PatchingDemoGameInstance.cpp. Add the following #includes at the top of the file under #include "PatchingDemoGameInstance.h":

   PatchingDemoGameInstance.cpp

   #include "PatchingDemoGameInstance.h"
   
   #include "ChunkDownloader.h"
   #include "Misc/CoreDelegates.h"
   #include "AssetRegistryModule.h"

   This will give you access to ChunkDownloader, and some useful tools for managing assets and delegates.

5. Declare the following function in PatchingDemoGameInstance.h under a protected header:

   PatchingDemoGameInstance.h

   void OnManifestUpdateComplete(bool bSuccess);

6. Create the following implementation for OnManifestUpdateComplete in PatchingDemoGameInstance.cpp:

   PatchingDemoGameInstance.cpp

   void UPatchingDemoGameInstance::OnManifestUpdateComplete(bool bSuccess)
   {
       bIsDownloadManifestUpToDate = bSuccess;
   }

   This function will be used as an asynch callback when the manifest update finishes.

7. Create the following implementation for the Init function in PatchingDemoGameInstance.cpp:

   PatchingDemoGameInstance.cpp

   void UPatchingDemoGameInstance::Init()
   {
       Super::Init();
   
       const FString DeploymentName = "PatchingDemoLive";
       const FString ContentBuildId = "PatcherKey";
   
       // initialize the chunk downloader
       TSharedRef<FChunkDownloader> Downloader = FChunkDownloader::GetOrCreate();
       Downloader->Initialize("Windows", 8);
   
       // load the cached build ID
       Downloader->LoadCachedBuild(DeploymentName);
   
       // update the build manifest file
       TFunction<void(bool bSuccess)> UpdateCompleteCallback = [&](bool bSuccess){bIsDownloadManifestUpToDate = bSuccess; };
       Downloader->UpdateBuild(DeploymentName, ContentBuildId, UpdateCompleteCallback);
   }

   Let's summarize what this code does:

   1. The function defines DeploymentName and ContentBuildID to match the values used in DefaultGame.ini. These are currently fixed values for testing, but in a full build you would use an HTTP request to fetch the ContentBuildID. The function uses the information in these variables to make a request to your web site for the manifest.

   2. The function calls FChunkDownloader::GetOrCreate to set up ChunkDownloader and get a reference to it, then stores it in a TSharedRef. This is the preferred way to get references to this or similar platform interfaces.

   3. The function calls FChunkDownloader::Initialize using the desired platform name, in this case, Windows. This example gives it a value of 8 for TargetDownloadsInFlight, which sets the maximum number of downloads that ChunkDownloader will handle at once.

   4. The function calls FChunkDownloader::LoadCachedBuild using the DeploymentName. This will check if there are already downloaded files on disk, which enables ChunkDownloader to skip downloading them a second time if they are up to date with the newest manifest.

   5. The function calls FChunkDownloader::UpdateBuild to download an updated version of the manifest file.

      • This is how the system supports update patches without requiring an entirely new executable.

      • UpdateBuild takes the DeploymentName and ContentBuildID alongside a callback that outputs whether or not the operation succeeded or failed.

      • It also uses OnManifestUpdateComplete to set bIsDownloadManifestUpToDate so that the GameInstance can globally recognize that this phase of patching is done.

   Following these steps ensures that ChunkDownloader is initialized and ready to start downloading content, and informs other functions of the manifest's status.

8. Create the following function implementation for UPatchingDemoGameInstance::Shutdown:

   PatchingDemoGameInstance.cpp

   void UPatchingDemoGameInstance::Shutdown()
   {
       Super::Shutdown();
   
       // Shut down ChunkDownloader
       FChunkDownloader::Shutdown();
   }

   Calling FChunkDownloader::Shutdown will stop any downloads ChunkDownloader currently has in progress, then clean up and unload the module.

3. Downloading Pak Files

Now that you have appropriate initialization and shutdown functions for ChunkDownloader, you can expose its .pak downloading functionality.

1. In PatchingDemoGameInstance.h, add the following function declaration for GetLoadingProgress:

   PatchingDemoGameInstance.h

   UFUNCTION(BlueprintPure, Category = "Patching|Stats")
   void GetLoadingProgress(int32& FilesDownloaded, int32& TotalFilesToDownload, float& DownloadPercent, int32& ChunksMounted, int32& TotalChunksToMount, float& MountPercent) const;

1. In PatchingDemoGameInstance.cpp, create the following implementation for the GetLoadingProgress function:

   PatchingDemoGameInstance.cpp

   void UPatchingDemoGameInstance::GetLoadingProgress(int32& BytesDownloaded, int32& TotalBytesToDownload, float& DownloadPercent, int32& ChunksMounted, int32& TotalChunksToMount, float& MountPercent) const
   {
       //Get a reference to ChunkDownloader
       TSharedRef<FChunkDownloader> Downloader = FChunkDownloader::GetChecked();
   
       //Get the loading stats struct
       FChunkDownloader::FStats LoadingStats = Downloader->GetLoadingStats();
   
       //Get the bytes downloaded and bytes to download
       BytesDownloaded = LoadingStats.BytesDownloaded;
       TotalBytesToDownload = LoadingStats.TotalBytesToDownload;
   
       //Get the number of chunks mounted and chunks to download
       ChunksMounted = LoadingStats.ChunksMounted;
       TotalChunksToMount = LoadingStats.TotalChunksToMount;
   
       //Calculate the download and mount percent using the above stats
       DownloadPercent = (float)BytesDownloaded / (float)TotalBytesToDownload;
       MountPercent = (float)ChunksMounted / (float)TotalChunksToMount;
   }

2. In PatchingDemoGameInstance.h, below your #includes, add the following dynamic multicast delegate:

   PatchingDemoGameInstance.h

   DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FPatchCompleteDelegate, bool, Succeeded);

   This delegate outputs a boolean that will tell you whether or not a patch download operation succeeded. Delegates are commonly used to respond to asynchronous operations like downloading or installing files.

3. In your UPatchingDemoGameInstance class, add the following delegate declaration under a public header:

   PatchingDemoGameInstance.h

   /** Delegates */
   
   /** Fired when the patching process succeeds or fails */
   UPROPERTY(BlueprintAssignable, Category="Patching");
   FPatchCompleteDelegate OnPatchComplete;

   These give you a place to hook into with Blueprint when a patching operation is finished.

4. Under a protected header, add the following declaration for ChunkDownloadList:

   PatchingDemoGameInstance.h

   /** List of Chunk IDs to try and download */
   UPROPERTY(EditDefaultsOnly, Category="Patching")
   TArray<int32> ChunkDownloadList;

   You will use this list to hold all the Chunk IDs that you want to download later. In a development setting, you would initialize this with a list of assets as-needed, but for testing purposes, you will simply expose the defaults so we can fill them in using the Blueprint editor.

5. Under a public header, add the following declaration for PatchGame:

   PatchingDemoGameInstance.h

   /** Starts the game patching process. Returns false if the patching manifest is not up to date. */
   UFUNCTION(BlueprintCallable, Category = "Patching")
   bool PatchGame();

   This function provides a Blueprint-exposed way to start the patching process. It returns a boolean indicating whether or not it succeeded or failed. This is a typical pattern in download management and other kinds of asynchronous tasks.

6. Under a protected header, add the following function declarations:

   PatchingDemoGameInstance.h

   /** Called when the chunk download process finishes */
   void OnDownloadComplete(bool bSuccess);
   
   /** Called whenever ChunkDownloader's loading mode is finished*/
   void OnLoadingModeComplete(bool bSuccess);
   
   /** Called when ChunkDownloader finishes mounting chunks */
   void OnMountComplete(bool bSuccess);

   You will use these to respond to async callbacks in the download process.

7. In PatchingDemoGameInstance.cpp, add the following implementations for OnDownloadComplete and OnLoadingModeBegin:

   PatchingDemoGameInstance.cpp

   void UPGameInstance::OnLoadingModeComplete(bool bSuccess)
   {
       OnDownloadComplete(bSuccess);
   }
   
   void OnMountComplete(bool bSuccess)
   {
       OnPatchComplete.Broadcast(bSuccess);
   }

   OnLoadingModeComplete will pass through to OnDownloadComplete, which will proceed to mount chunks in a later step. OnMountComplete will indicate that all chunks have finished mounting, and the content is ready to use.

8. In PatchingDemoGameInstance.cpp, add the following implementation for PatchGame:

   PatchingDemoGameInstance.cpp

   bool UPGameInstance::PatchGame()
   {
       // make sure the download manifest is up to date
       if (bIsDownloadManifestUpToDate)
       {
           // get the chunk downloader
           TSharedRef<FChunkDownloader> Downloader = FChunkDownloader::GetChecked();
   
           // report current chunk status
           for (int32 ChunkID : ChunkDownloadList)
           {
               int32 ChunkStatus = static_cast<int32>(Downloader->GetChunkStatus(ChunkID));
               UE_LOG(LogTemp, Display, TEXT("Chunk %i status: %i"), ChunkID, ChunkStatus);
           }
   
           TFunction<void (bool bSuccess)> DownloadCompleteCallback = [&](bool bSuccess){OnDownloadComplete(bSuccess);};
           Downloader->DownloadChunks(ChunkDownloadList, DownloadCompleteCallback, 1);
   
           // start loading mode
           TFunction<void (bool bSuccess)> LoadingModeCompleteCallback = [&](bool bSuccess){OnLoadingModeComplete(bSuccess);};
           Downloader->BeginLoadingMode(LoadingModeCompleteCallback);
           return true;
       }
   
       // we couldn't contact the server to validate our manifest, so we can't patch
       UE_LOG(LogTemp, Display, TEXT("Manifest Update Failed. Can't patch the game"));
   
       return false;
   
   }

   This function goes through the following steps:

   1. First, it checks if the manifest is currently up to date. If you have not initialized ChunkDownloader and successfully gotten a fresh copy of the manifest, bIsDownloadManifestUpToDate will be false, and this function will return false, indicating a failure to start patching.

   2. Next, if the patching process can continue, the function gets a reference to ChunkDownloader. It then iterates through the download list and checks the status of each chunk.

   3. Two callbacks are defined:

      • The DownloadCompleteCallback will be called when each individual chunk finishes downloading, and it will output a message when each of them successfully downloads or fails to download.

      • The LoadingModeCompleteCallback will fire when all chunks have been downloaded.

   4. The function calls FChunkDownloader::DownloadChunks to begin downloading desired chunks, which are listed in ChunkDownloadList. This list must be filled with the chunk IDs you want before calling this function. It also passes the DownloadCompleteCallback.

   5. The function calls FChunkDownloader::BeginLoadingMode with the callback you defined earlier.

      • Loading Mode will tell ChunkDownloader to start monitoring its download status.

      • While chunks can download passively in the background without calling Loading Mode, using it will output download stats, enabling you to create a UI that can track download progress for the user.

      • You can also use the callback function to run specific functionality when an entire batch of chunks is downloaded.

9. In PatchingDemoGameInstance.cpp, add the following implementation for OnDownloadComplete:

   PatchingDemoGameInstance.cpp

   void UPatchingDemoGameInstance::OnDownloadComplete(bool bSuccess)
   {
   if (bSuccess)
       {
           UE_LOG(LogTemp, Display, TEXT("Download complete"));
   
           // get the chunk downloader
           TSharedRef<FChunkDownloader> Downloader = FChunkDownloader::GetChecked();
   
           FJsonSerializableArrayInt DownloadedChunks;
   
           for (int32 ChunkID : ChunkDownloadList)
           {
               DownloadedChunks.Add(ChunkID);
           }
   
           //Mount the chunks
           TFunction<void(bool bSuccess)> MountCompleteCallback = [&](bool bSuccess){OnMountComplete(bSuccess);};
           Downloader->MountChunks(DownloadedChunks, MountCompleteCallback);
   
           OnPatchComplete.Broadcast(true);
   
       }
       else
       {
   
           UE_LOG(LogTemp, Display, TEXT("Load process failed"));
   
           // call the delegate
           OnPatchComplete.Broadcast(false);
       }
   }

   This is another complex function, so we will break down what it is doing. This runs when your .pak files have been successfully downloaded to a user's device.

   1. First, it gets a reference to ChunkDownloader.

   2. Next, the function sets up a Json array and fills it with information from ChunkDownloadList. This will be used to make your request.

   3. The function uses MountCompleteCallback to output whether or not the patch was successfully applied.

   4. The function calls ChunkDownloader::MountChunks using the Json list and MountCompleteCallback to start mounting the downloaded chunks.

   5. If the download was successful, the function activates the OnPatchComplete delegate with a value of true. If it wasn't successful, it activates with a value of false. UE_LOG outputs error messages according to the point of failure.

4. Setting Up a Patching Game Mode

To initiate the patching process, you can make a level and game mode specifically to call PatchGame and output patching stats to the screen.

1. In Unreal Editor, create a new Blueprints folder in the Content Browser. Then, create a New Blueprint using PatchingDemoGameInstance as the base class.

   Click image to enlarge.

   Name the new Blueprint class CDGameInstance.

   Click image to enlarge.

   You will use this Blueprint as a more accessible way to edit settings and track the chunk download process.

2. Create a new Game Mode Blueprint called PatchingGameMode.

3. Create a Maps folder, then create two new levels called PatchingDemoEntry and PatchingDemoTest. The Entry level should be based on an empty map, and the Test level should be based on the Default map.

   Click image to enlarge.

4. In the World Settings for PatchingDemoEntry, set the GameMode Override to PatchingGameMode.

   Click image to enlarge.

5. Open your Project Settings and navigate to Project > Maps & Modes. Set the following parameters:

   Click image to enlarge.

   ID	Parameter	Value
   1	Game Instance Class	CDGameInstance
   2	Editor Startup Map	PatchingDemoTest
   3	Game Default Map	PatchingDemoEntry

6. Open CDGameInstance in the Blueprint editor. In the Defaults panel, add three entries to the Chunk Download List. Give them values of 1001, 1002, and 1003. These are our Chunk IDs from our three .pak files.

7. Open PatchingGameMode in the Blueprint Editor and navigate to the EventGraph.

8. Create a Get Game Instance node, then cast it to CDGameInstance.

9. Click and drag from As CDGameInstance, then click Promote to Variable to create a reference to our game instance. Call the new variable Current Game Instance.

10. Click and drag from the output pin of Set Current Game Instance, then create a call to Patch Game.

11. Click and drag from the Return Value of Patch Game, then click Promote to Variable to create a boolean in which to store its value. Call the new variable Is Patching In Progress.

    Click image to enlarge.

12. Create a Get Current Game Instance node, then click and drag from its output pin and create a call to Get Patch Status.

13. Click and drag from the Return Value pin of Get Patch Status, then create a Break PatchStats node.

14. Click and drag from the Tick event and create a new Branch node. Attach Is Patching In Progress to its Condition input.

15. Click and drag from the True pin on your Branch node, then create a Print String node. Use BuildString (float) to output the Download Percent from the Break PatchStats node. Repeat this step for Mount Percent as well.

    Click image to enlarge.

16. From the Print String node, create a Branch node, then create an AND node and connect it to the Condition pin.

17. Create a Greater Than or Equal To node to check if Download Percent is 1.0 or higher, then do the same thing for Mount Percent. Connect both of these to the AND node. If both of these conditions are true, use Open Level to open your PatchingGameTest level.

    Click image to enlarge.

Now when your game runs, it will open the Entry map, run ChunkDownloader, and output the progress on downloading and mounting the chunks in your Chunk Download list. When the download finishes, it will then transition to your test map.

5. Displaying the Downloaded Content

To display our character meshes, you will need to get references to them. This will require Soft Object References as you need to verify that your assets are loaded before you use them. This section will walk you through a simple example of how to spawn actors and fill their skeletal meshes from soft references.

1. Open the PatchingDemoTest level, then open the Level Blueprint.

2. Create a new variable called Meshes.

   • For its Variable Type, choose Skeletal Mesh.

   • Hover over the entry in the types list and select Soft Object Reference. This will change the color of the variable from blue to soft green.

   Click image to enlarge.

   Soft Object References are a type of smart pointer that can safely reference ambiguous assets. We can use this to check if our mesh assets are loaded and available before using them.

3. Click the icon next to the variable type of Meshes to change it to an Array. Compile your Blueprint to apply the change.

   Click image to enlarge.

4. In the Default Value for Meshes, add three entries and select the skeletal meshes for Boris, Crunch, and Khaimera.

   Click image to enlarge.

5. In the EventGraph for the level, click and drag from the BeginPlay event, then create a For Each Loop and connect it to your Meshes array.

6. Click and drag from the Array Element pin on your For Each Loop, then create an Is Valid Soft Object Reference node. Create a Branch from the Loop Body and connect it to the Return Value.

7. Create a Spawn Actor From Class node and connect it to the True pin for the Branch node. Choose Skeletal Mesh Actor for the Class.

8. Click and drag from the Array Index in the For Each Loop and create an Integer x Float node. Set the float value to 192.0.

9. Click and drag from the return value of the Integer x Float node to create a Vector x Float node, and give the Vector a value of (1.0, 0.0, 0.0).

   • This will make a coordinate 192 units away from the origin for each time we go through the For Each loop. This will give each of our meshes some space when we spawn them.

10. Use the vector from the previous step as the Location in a Make Transform node, then connect the Return Value to the Spawn Transform input of the Spawn Actor node.

11. Click and drag from the Return Value of the Spawn Actor node, then get a reference to its Skeletal Mesh Component. Use that to call Set Skeletal Mesh.

12. Click and drag from the Array Element node, then create a Resolve Soft Object Reference node. Connect the output of this node to the New Mesh input pin for Set Skeletal Mesh.

    Click image to enlarge.

13. Move the Player Start inside the level to (-450, 0.0, 112.0).

    Click image to enlarge.

14. Save your progress and compile your Blueprints.

When the level loads, the skeletal meshes for each of your characters will spawn. If the soft object reference does not work, then the chunks for each character are not yet mounted, their assets will not be available, and they will not spawn.

6. Testing Your Game

Finally, you need to test your project in a standalone build. Pak mounting does not work in PIE mode, so this is a necessary step to test your patching functionality.

1. Package your project.

2. Copy the .pak files and manifest to corresponding folders on your IIS test website.

3. Make sure the IIS process and website are both running.

4. Run your packaged executable.

End Result

You should see a black screen with the patching output in the upper-left side of the screen, then, when both the patching and mounting status reach 100%, your game should load into the default map and display Boris, Crunch, and Khaimera. If something goes wrong with the patching or mounting process, none of them will appear.

On Your Own

From here, there are several next steps you can take to flesh out your chunk download scheme:

• Build a UI that appears during loading mode and displays progress bars and prompts for the player.

• Build UI prompts for errors such as timeouts and installation failures.

• Create a custom subclass of PrimaryAssetLabel to include additional metadata about your assets. For example, Battle Breakers' custom PrimaryAssetLabel class includes a Parent Chunk that must be loaded as a prerequisite to use the current chunk.