Content Browser Database

Overview

The Content Browser uses a special database backend for storing assets, tags and collections.

We call this system the Game Asset Database (or GAD for short), but it's really a mixture of several database components: A journal server, a checkpoint file, thumbnail storage and an in-memory database. There is also a checkpoint commandlet that is used to update and maintain the database.

All of this is pretty much transparent to a user working in the Content Browser -- but it adds some nice features.

Why do I need a game asset database?

The game asset database certainly adds complexity, but there are important benefits to this system:

• All game content is instantly available for browsing, whether it's loaded or not.
• Users never have to check out packages to tag assets or add them to collections.
• Thumbnails are displayed for all game assets, even assets that aren't loaded.
• All tag data is still properly source-controlled (e.g. roll backs are possible.)

In the editor, most common interactions with non-loaded assets are supported. The editor will simply load the asset on demand before the action is executed. For example, if the user drags and drops an unloaded static mesh into the 3D viewport the Content Browser will automatically load the mesh before placing it.

Content Browser Database Interaction

When the editor starts up it first loads all asset names, tags and collections from a read-only checkpoint file. This file represents the state of all tags in the game at the time the last checkpoint process was run (at least once per day.) Next the editor connects to the journal server and downloads any recently-applied tags.

The Content Browser now knows about all assets, packages, groups, tags and collections in the entire game. The user can browse all game assets (loaded or not) using the graphical interface. Thumbnails for loaded assets will be generated dynamically, but thumbnails for non-loaded assets will be quickly loaded straight from their asset package files.

When the user decides to tag an asset, the Content Browser updates it's in-memory database with the change, then immediately connects to the journal server and uploads the tag command. Note that the checkpoint file is never modified by the Editor -- all changes are routed to the journal server and resolved later on during by the checkpoint commandlet.

The following chart shows how the Content Browser interacts with the game asset database.

Additionally, it's possible to run the editor in "Offline Mode" which allows for limited functionality without using a live server. See the section near this bottom of this page about "Offline Mode" for more information about that.

Build System Database Interaction

The following chart shows how a script running on a build machine generates and maintains asset database data. It seems complicated at first but it's actually very straightforward and easy to maintain.

Database Setup

Here is a brief outline of the steps required to setup the Game Asset Database as well as the checkpointing process:

• Run a commandlet to create a new Checkpoint file for your game.

• Setup a regular build process to 'checkpoint' the asset database and update the file in source control. (We run this automatically every night.)

• Install and configure the journal server.

• Set the database/branch settings in each game's editor configuration file.

• Optionally, resave all asset packages to automatically generate thumbnails (only if necessary)

This seems like a lot of work...

The Content Browser can operate without a checkpoint file or journal server. We don't recommend it, but here's is what you'll end up with:

Without a checkpoint file:

• Users won't be able to browse the entirety of the game's content. Only loaded assets will be searchable and interactive.
• Some types of meta data won't be available (e.g. the date an asset was added.)
• The experience will be similar to that of the Generic Browser, except you'll still see all packages and directories.

It's pretty easy and fast to create a checkpoint file, even locally, so you shouldn't have to resort to that.

Without a journal server:

• Users won't be able to tag assets or search by tags (except in Offline Mode)
• Users can't place assets into collections, even private collections (except in Offline Mode.)
• Users can't share tags and collections with others
• Various "live" features won't be available

No way you'd want to give up all those goodies, right? Totally worth the effort! Read on to learn about how to get up and running.

Create a checkpoint file

First you'll need to create a initial checkpoint file for each of your games.

Games have a single GAD checkpoint file stored as /MyGame/Content/GameAssetDatabase.checkpoint. This is a binary file that stores meta data for all assets in the entire game (including all engine assets.) The editor will load the checkpoint file into memory at startup where it will be used by the Content Browser to locate assets quickly.

It's easy to create a checkpoint file:

• Make sure you're synced up to the latest game content
• Run the checkpoint commandlet:
  • MyGame.exe CheckpointGameAssetDatabase
• (It will take a few minutes as it digests all of the game content.)
• That's it! You now have a working checkpoint file for this game.
  • If you load up the editor you'll now be able to browse all assets in the game using the Content Browser.
• Normally, you should check this file into source control. You can read more about that in the next section.

The checkpoint commandlet doesn't usually need any extra parameters, but there are some options you may use from time to time. See the section below for more info.

Regularly checkpointing game content

There is some flexibility as to how you handle this part. Here's how we have things setup at Epic:

• Every night around 3:00 AM, a machine running our build system (Controller) executes a build script (CheckpointGAD.build) that performs the following:
  • For each game (or game branch):
    • Syncs to the last successful engine build (including binaries). Often, the last good build had just completed immediately prior.
    • Syncs all game content (and only game content) to the head revision. This is so that the asset checkpoint will be as up-to-date as possible, even if we're running a slightly older build of the engine.
    • Checks out the game's GAD checkpoint file (GameAssetDatabase.checkpoint)
    • Executes the checkpoint commandlet to update the checkpoint file.
      • MyGame.exe CheckpointGameAssetDatabase
      • The commandlet's output is checked for error text by the build system engine. If something went wrong, the checkpoint fails and the file is reverted, and emails are sent to appropriate folks.
    • Submits the updated GAD checkpoint file (GameAssetDatabase.checkpoint)

Now, team members will sync the file each day along with the rest of the game's content and have access to the entirety of game assets in the state they were checkpointed in. Note that there's no harm in running this process even more frequently than once per day if needed. (We sometimes manually trigger GAD checkpoints in our build system during the day.)

Setting up the journal server

This involves installing a SQL server and configuring it so your developers can read and write to it.

The Content Browser demands on SQL are extremely light so we recommend using the free 'Express Edition' of Microsoft SQL Server, but if you're already using the full version of MS SQL that will work great, too. Remember, the server is only used as a scratchpad for newly-tagged assets and usually contains no more than a few thousand rows at any point in time. Most of the actual GAD data is stored in a 'Checkpoint File' on disk.

Note: If your team has no need for asset tags, collections and certain types of asset meta data (e.g. Date Added), you could skip the Journal server setup and rebuild the checkpoint file from scratch whenever you want to freshen your asset database. It's up to you.

Installing MS SQL Server

(Note: You can skip this step if the computer that's going to host your journal server already has a version of MS SQL Server install already. For example, if your team has already purchased a copy of the unrestricted version of MS SQL Server you should probably use that instead.)

We recommend you install the free version of MS SQL Server Express Edition. (We used the 2008 version, but 2005 should also work.)

• The installer may warn you about some prerequisites that must be installed first (e.g. Windows Installer, Windows Powershell.) Go ahead and install those first if prompted.

• For MS SQL Server Express 2008, you can use the default installation settings the entire way through. If your using a different version of the application, make sure that MS SQL Server Management Studio is installed along with the server so you can easily edit/audit settings.

• If prompted for a Server Admin name you can you your own user name, or check with your IT department for what to use there.

• You can name the database whatever you'd like, just take note of it because we'll need that name later on when configuring Unreal Editor.
  • The examples here will use the default database name: SQLEXPRESS

Automatic journal server setup (recommended!)

• Use the CreateContentJournal.sql script to create a journal server instance
  • The script can be found in the UE3 source drop: /Development/Src/Engine/SQL/CreateContentJournal.sql
  • Use the sqlcmd.exe command-line application to execute this script.
    • You'll need to pass in the name of the computer running the server along with the server's name (unless it's the default server instance.)
    • You'll also pass in the path and file name of the CreateContentJournal.sql file from UDN
    • For example: sqlcmd -S Computer_Name\Server_Name -i CreateContentJournal.sql
  • If everything goes well you won't see any error messages, and your journal server is ready to rock!

• Note: Normal Windows authentication should be enabled by default on your new server, but you may need to talk to your IT department to make sure everything is configured such that your team member's computers can communicate with it directly.

• If for some reason you need to create the server instance manually, follow the steps in the next section.

Manual journal server setup (only if needed)

You only need to follow these steps in the unlikely case that the CreateContentJournal.sql script didn't work for you. If you ran the script successfully, the database is already created and you can skip this entire section!

• Load up MS SQL Server Management Studio and connect to your database.

• In the Object Explorer on the left, right click no "Databases" and click "New Database...". The "New Database" dialog window will pop up.
  • Set the "Database name" to: ContentJournal
  • Click "OK" to close the dialog and save the new database.

• Under "Databases" in the Object Explorer, expand "ContentJournal" so that you can see it's compnents (Tables, Views, etc.)

• Under "ContentJournal" in the Object Explorer, right click on "Tables" and click "New Table...". The table designer will open up in the content area of the application.
  • Configure the table's columns as such:
    • First column:
      • Set "Column Name" to: DatabaseIndex
      • Set "Data Type" to: int
      • Set "Allow Nulls" to: No (Unchecked)
    • First column's properties:
      • Set "Identity Specification -> (Is Identity)" to: Yes
    • Right click on the first column ("DatabaseIndex") and click "Set Primary Key"
    • Second column:
      • Set "Column Name" to: Text
      • Set "Data Type" to: nvarchar(MAX)
      • Set "Allow Nulls" to: No (Unchecked)
  • Press Ctrl+S to save the new table. The save dialog will pop up.
    • Save the new table with the following name: Entries

• Great! The journal server is now alive and kicking.

• Note: Normal Windows authentication should be enabled by default on your new server, but you may need to talk to your IT department to make sure everything is configured such that your team member's computers can communicate with it directly.

Configure Unreal Editor

Okay, you're almost done!

After the journal server is setup, you'll need to tell UnrealEd where to find it. Also you need to set the "branch name" for the particular source control branch of the game the editor is running from.

• Open BaseEditor.ini in a text editor. (/Engine/Config/BaseEditor.ini)
  • If you need a game-specific settings, you can make these changes in /MyGame/Config/DefaultEditor.ini instead

• Locate the [GameAssetDatabase] section in the .ini file. If it doesn't exist yet, then go ahead and create this section.

• Set JournalServer to the path to the journal SQL server (Computer_Name\SQL_Server_Name)
  • For example: JournalServer=ServerComputer\MYSERVERNAME
  • If your server is the default SQL server instance on the remote computer, then you can omit the server name
    • For example: JournalServer=ServerComputer

• Set BranchName appropriately for your game's depot path.
  • For example: BranchName=Main
  • You can use whatever name for your branch you think is appropriate, but you must use only alpha-numeric characters.
  • Try to keep the branch name somewhat short as it will be attached to every single journal entry.

• While we're on the subject of .ini settings, there are a few user preferences that can be set for the Content Browser to allow or disallow creation and deletion of tags and collections. You should have your art leads and tech artists enable tag creation by setting the following:
  • In MyGameEditorUserSettings.ini, add a section for [ContentBrowserSecurity]
  • Under this section, set bIsUserTagAdmin=True and bIsUserCollectionsAdmin=true
  • Note that this isn't really supposed to be a security system -- it just helps to prevent people from accidentally destroying data.

Important: The branch name must be set appropriately in all source control branches of your game, otherwise asset and tag data will collide between branches (which can be really, really bad.) If you branch your game and have artists/LDs working in either branch, it's critical to update the BranchName in the new branch.

Generating thumbnails for assets

When the Content Browser needs to display a thumbnail for an asset that isn't currently loaded, it will check the asset's package to see if it has a thumbnail and quickly loads then displays it. This happens without actually loading the asset's package or any of it's data.

Thumbnails are generated automatically for most asset types by UnrealEd. You simply need load and save the package and the editor will make sure that thumbnails are rendered for all of the supported asset types.

If you're seeing a lot of assets in the Content Browser missing thumbnails, it probably means that the packages need to be resaved in UnrealEd. Note that there are many asset types that don't support custom thumbnails yet (e.g. Sounds, CameraAnims, etc.) Also note that you must save the packages in the editor to update the thumbnails. Commandlets (such as ResavePackages) are unable to generate thumbnails as a renderer is required for this.

For loaded assets, the editor will generate their thumbnail in real-time (we don't load them from a package file.)

All thumbnails are cached in memory for a short while to improve performance, but most are evicted quickly to keep memory usage light.

Advanced topics

Offline Mode

It's possible to run the Content Browser in "Offline Mode", where no SQL connection is required but you can still make use of tagging and collections.

To use Offline Mode, set the following in your EditorUserSetings.ini file:

[GameAssetDatabase] OfflineMode=true

Now, the editor won't bother trying to connect to a SQL database at startup. Instead it will create a journal file on disk where your tagging and collection actions will be stored.

When working with tags and collections from a different PC, you'll need to make sure that your user name matches up. For example if you're working from home through a VPN connection and want to interact with your private collections that you created at work, you'll need to set the "override user name" in your preferences file.

To set the "override user name", open EditorUserSettings.ini and set the following:

[GameAssetDatabase] OverrideUserName=MyUserName

Note that in Offline Mode, you won't be able to share your tags or collections with other users. However, if you eventually disable Offline Mode after a SQL database is available, the editor will automatically upload your accumulated tagging changes.

Offline Journal Alarm

The first time the Editor is run, the date field "UserJournalUpdateAlarm" is added to UTEditorUserSettings.ini and set to 7 days in the future. This represents the next time the alarm will go off (Note - this is saved when the app closes).

To test:

• Set the time for this entry to a date prior to your next execution time to ensure the alarm will go off.
• Restart the Editor.
• Upon startup, you will see the following dialog:

• "Check Now" will run the check point updating process, make a back up of the journal file, delete the original journal file, and set the alarm to 7 days in the future.
• "Next Run" does nothing as the alarm time is properly set to go off for the next run.
• "Wait a Day" and "Wait a Week" set the alarm time forward the expected amount of time from the current time.

This can be enabled for Online builds by setting "[GameAssetDatabase]OfflineMode" to TRUE in UTEditor.ini and ensuring that "[GameAssetDatabase]UseJournalUpdateAlarm" is TRUE (it's the default). This should only be used for testing, as Epic and licensees should be using a content server. Similarly, setting "[GameAssetDatabase]UseJournalUpdateAlarm" to FALSE will disable the Asset Database Maintenance feature.

Checkpoint commandlet options

The commandlet has some additional options that you really shouldn't need to use except when debugging problems:

• -NoDeletes tells the commandlet not to delete any expired journal entries from the SQL server. Any entries that are older than the built-in threshold will be left in the database. Note that if you always use this option your SQL database will grow very large and your editor startup performance will diminish. The next time you run the checkpoint process without this argument it will go and the delete expired assets as per usual.
• -ImportOldTags (Removed in May 2009 QA Build, changelist 33172) Imports legacy Content Tag data from all packages and merges that into the game asset database. See the section below on importing legacy content tags for more information.
• -PurgeGhosts forces the commandlet to immediately expire (delete) any Ghost asset entries from the database. Normally ghost assets are automatically expired after a fixed number of days, unless the asset materializes in a package before then.
• -Repair will attempt to repair any errors in the database. Note that for errors, a repair constitutes eradicating the problematic entry from the database.
• -Reset will build a fresh checkpoint file from scratch. It will ignore all of the existing tags, collections, journal server data and meta data and wipe the slate clean. You probably shouldn't use this unless you know what you're doing.
• -Verify will only check the state of the database and report warnings. It won't actually change anything or save state.
• -DeleteCollections will delete both public and private collections

Importing legacy content tags

The old Content Tag Browser and all related functionality, including the data in package files, was removed from UE3 in the May 2009 QA Build (changelist # 333172.) If you already have a lot of content tags set on your assets and want to convert those to the new system, you'll have to follow the steps below:

• Be careful not to resave any content packages using the May 09 QA build (changelist 333172) or later yet! Resaving packages will destroy your content tags.
• Either sync to the April 09 QA build, or sync the May 09 QA build except changelist 333172.
• For each game, run the GAD commandlet to convert old content tags:
  • MyGame.exe CheckpointGameAssetDatabase -ImportOldTags
• This will create a file (GameAssetDatabase.checkpoint) with all of your tags
  • The Content Browser always loads tag data from this file
• Now you can sync up to a later QA build (all changelists) and you're good to go!

Unverified assets and Ghosts

From time to time you may see an asset in the Content Browser visually marked as Unverified. This just means that the asset's information was downloaded from the journal server and has not yet been "blessed" by the checkpoint commandlet yet.

When the checkpoint process runs it will download the unverified assets from the journal server, then check to make sure these assets actually exist in game content. (Usually, your build system is performing the checkpointing, so this would mean the assets were submitted to source control.)

Assuming the asset was found by the checkpoint commandlet, the next time a user syncs to the latest checkpoint file, the assets will no longer be marked as unverified.

If, however, the asset wasn't found (because perhaps a user hasn't checked in a newly-created asset yet, or maybe never even saved it at all), the checkpoint tool will mark the asset as a Ghost. Ghost assets don't exist on disk so are never displayed in the Content Browser, so most users will never even know about this feature.

Asset ghosts are still kept around in the database for a fixed number of days in case a user does end up checking in the asset. This is important because the asset may already have tags or other metadata assigned to it, even though it hasn't been submitted to source control yet. Eventually the ghost will expire and be deleted from the database.

If you never want to see other user's Unverified assets in the editor you can use the -NoLiveTags command-line parameter which tells the editor to only download information about assets that you created.