Hot Spot Report Generation

Overview

The below article explains how to create "Hot Spot Reports" for your level designers. These reports will be created by your QA department and are useful in identifying memory dips due to sub-levels streaming in and out. They will allow your LDs to better see which buckets are costing the most memory, and should allow them to optimize their levels to better fit in memory.

Creating Raw MemLeakCheck Data

To get the raw data that we will use to plot the memory status for SP levels you will need to do the following:

1. Load up UnrealConsole, and connect to the instance of the game that will be used for this test.
2. Launch into DebugConsole executable in the build you intend to capture from.
3. Load into the SP level you intend to do a capture for.
4. Once in the level use the following command:
   DoMemLeakchecking 30

   The 30 denotes the number of seconds between memleakcheck captures. You can increase or decrease the frequency as necessary, but generally you will want to keep this number consistent in order to properly track trending over time.

5. Turn on stat levels to be able track when the P-level is going to change.
6. Progress through the entire P-level and transition to next P-level.
7. After 2 captures into the P-level that you transition into back out to the main menu and/or reboot.
8. Always reboot before starting another level capture.
9. The memleakcheck files will be created in the following folder, depending on the platform being profiled:
   • For PC/UDK:
     [install]\[GameName]\Profiling\MEMLEAKS
   • For iOS (after backing up the documents directory to PC. See Deployment Tools for info on backing up the documents directory.):
     [install]\[GameName]\iOS_Backups\[DeviceName]\Profiling\MemLeaks
10. Each memleakfile will be parsed into a folder named after the level it was started in.
    For example, if you began a capture in level-a, and progressed into level-b and level-c captures for this entire run would be in the level-a folder.
11. Copy these memleakcheck files/folders into a location that can be reached easily.

Creating CSVs To Pull Data From

1. Launch the memleakcheckdiffer.exe from the following folder:
   ..\Binaries\
2. In the MemLeakCheckDiffer, Choose File > Open all Memleak files in a folder and sub-folders
3. Browse, and Navigate to the folder where you placed your memleakcheck files.
4. Wait for the folders to populate the MemLeakCheckDiffer.
5. Once all of the folders have populated MemLeakCheckDiffer window, choose to Generate Overall Report.
6. CSVs for each level will be generated, and a GlobalSummary CSV will also be created.
7. CSVs are located inside the folder that your MemLeakCheck files were placed.

Parsing For Overall Trending Data

1. Open the GlobalSummary spreadsheet in Excel.
2. To more easily visualize the content data, hide all of the rows outside of the following:
   • LowestTFP
   • AnimSequenceClasses
   • SkeletalMesh_Classes
   • SoundNodeWave
   • StaticMesh_Classes
3. Highlight these 5 rows starting with the bucket name and ending with the last value in the row.
4. After they are selected, choose to Insert > Line Graph.
5. This will give you a quick glimpse of the state of each of the content buckets as well as overall memory trending on the level.

Parsing Individual Level CSVs

After you have an idea of where the dips are you will need to work to figure out which content buckets grew to cause the memory change to happen. This is a matter of finding the lowest memory value from a particular dip, and taking the values of the buckets from the dip and subtracting from it the value of the buckets before the dip began. This will allow you to easily tell which content buckets caused the most problems at this location.

For example if the level had 10 MB of AnimSequence Class data before a dip, and once the dip begins it increased to 25 MB in this bucket the LD and other devs will know that something that required a lot of animation data was loaded here, and this to cause the level to go low on memory.

This data should be included along with each dip whenever possible. Simply letting the LD know how much the content buckets changed along w/ the sub-levels that were loaded when a dip occurs can often be enough to allow them to work out what needs to be shifted around or what might need to be trimmed from the level.

Preparing 'Human-Friendly' Hot Spot Reports For Level Designers

When preparing a hot spot report for a level designer you need to keep in mind that you want to deliver as much actionable information as possible. This means they will need to know what was going on when the memory dipped. To this end we make sure to include a list of the new sub-levels that streamed in when the memory began to go lower.

We also provide a bugit location that can take the LD directly to the area of the level before the dip occurs in order to allow them to look around at this location in the editor or in game to make they are familiar with what is going on here and how much might need to be changed.

Below is an example of our hot spot reports:

BugItGo -1858.2000 -2162.7903 1572.0010 64073 -16686 0=

BugItGo 904.3768 261.3029 1761.1804 -4788 -20256 72

Items To Be Called Out

Any large changes in a single bucket where a dip occurs is usually a good thing to call out. In many cases things like cinematics will cause certain buckets to go up a large amount because of bugs that are simple fixes like turning bake and prune on. It is always good to get all of the low hanging fruit such as this as quickly as possible.

Items To Avoid

When compiling a hot-spot report you will want to avoid calling out buckets that the LDs can't have an effect on. You also want to determine if there are any buckets that aren't giving actionable data. In the case of our tests Texture pool stats were not a valid bucket to look at for memory because of the way they were capped was different than what was being recorded.

Preparing A Followup Hot Spot Report

Followup reports allows LDs to see how much improvement or decline has happened since they received the first memory report. It is important to send out a follow-up report when memory has improved (even when the memory is good enough that another report isn't necessary) to help the LD understand the changes they are making are having a positive impact.

The ultimate goal is to get and keep the levels in memory, and having LDs looking forward to seeing how their level currently stacks up is a big part of making that happen.

Long Duration Runs

Once enough levels are in memory it is time to begin doing longer duration memory runs on multiple levels. To do this you merely need to find a large stretch of levels without blockers and start the domemleakchecking 30 command and play as far as possible. We run with reducepoolsize during these runs so we can get the longest stretch possible.

Using this data you can compare the difference between the individual level runs and continuous runs to get a basic memory delta between the two. Looking at this delta will give you an idea of how much overhead you will need to start building into individual levels, and will help to locate any memory leaks that may be happening.

Cold Spot Reports

Once memory dips have been resolved, and the game is within memory constraints (and if time permits) a cold spot report system can be implemented. What this means is that the same process to gather memory over time is done, but now you will want to concentrate on what areas have room to add rather than need to cut back. Done early enough this will allow LDs to flesh out areas better that are well within memory constraints.