UDN
Search public documentation:

MemoryDebugging
日本語訳
中国翻译
한국어
Interested in the Unreal Engine?
Visit the Unreal Technology site.

Looking for jobs and company info?
Check out the Epic games site.

Questions about support via UDN?
Contact the UDN Staff

UE3 Home > Memory System > Memory Debugging

Memory Debugging

Memory Debugging Overview

This section will help you to understand how to approach and handle memory issues – in particular, for consoles that limit your memory usage to the physical size of memory.

This happens pretty often during game development due to situations such as unplanned budget, giant assets, code bugs, and more.

This will provide a basic idea of how to handle many of these situations, but please refer to platform-specific issue for more details.

Debugging memory: Step by step

1. Load the Game

If you can't load game due to out of memory, please refer to the Level Optimization page to find out how to optimize level.

It is generally a good idea to have established guidelines for a memory budget – for example, how many Static Meshes you can use in a level; or other assets such as Skeletal Meshes, Animations or Sounds. This can vary dramatically based on what your game does and what it is about.

2. Run the Game and Observe

Even if the game runs, it doesn't necessarily mean you're safe from running out of memory. You need to know how much memory your game is using and how much is available to maximize, while avoiding overdoing it.

Platform providers have tools to do this. Please refer platform section below.

3. Reduce memory usage (no OOM)

When you're running out of memory, you need to narrow it down to what that could be.

Some possibilities include:

  • The level that has too many Static Meshes.
  • AIs that create too many projectiles and particles.
  • Code that allocates too much memory.

Basic Understanding

There is no universal answer as to how much memory should go to where.

It is up to your game and where your game decides to spend.

So you need to understand where your current usage is and you need to find the right budget.

  • Budget: Decide what to use memory on.
  • Make sure all assets are optimal and don't have unnecessary references.
  • Make sure you have enough buffer (for fragmentation) to keep your game running.

Case References 

           #define KEEP_TRACK_OF_NOVODEX_ALLOCATIONS 1

Case References

Level Optimization

Stats

STAT LEVELS

Run Stat levels to see how many levels are loaded on memory.

  • Green: Unloaded
  • Red: Loaded (with the time it took to load)
  • Blue: Unloaded and being garbage collected
  • Yellow: Loaded but not visible
  • Pink: Prepare to be loaded

If you have a level that doesn't have to be loaded, please make sure to not load it. Yellow levels are good candidates to not load or to defer if you can. If you have a big streaming level, try to split it or optimize.

STAT MEMORY:

Stat Memory provides basic outline of assets' memory usage in the level

  • Audio Memory
  • Novodex Allocation
  • Animation
  • Vertex Lighting
  • StaticMesh Vertex/Index
  • SkeletalMesh Vertex/Index
  • Decal Vertex/Index
  • VertexShader
  • PixelShader
  • Texture Pool Size
  • FaceFX

In many cases, a lot of these such as Animation, SkeletalMesh, Audio, and FaceFX, are co-related to the game data because of Unreal reference system. That is elaborated on further in this document.

Other Useful Console Commands:

  • ListSpawnedActors
  • ListLoadedPackages
  • ListPrecacheMapPackages
  • OBJ BULK

See the Level Optimization page for details on optimization; and the Stats Descriptions page for more stats descritpions.

Engine Configuration

Texture Pool Size:

All textures except render targets use texture pool. This includes lightmaps. (Using lightmaps instead of vertex lighting is a clear win on memory savings because they won't fluctuate system memory.)

Also make sure you have an optimal texture pool size for your game. Stat Memory will show how much of texture pool is being used. Based on that, you'll be able to determine the best number for your game. 

In *Engine.ini,
      [TextureStreaming]
         PoolSize=120

Frequent Garbage Collection:

If your game needs more frequent garbage collection, please use this configuration. 

In *Engine.ini,
      [Engine.Engine]
         TimeBetweenPurgingPendingKillObjects=10 (every 10 seconds)

Animation Optimization

When an AnimSet is referenced by the game, it will load all animations in the AnimSet. This can be waste if you only use a part of them. If all animations in the AnimSet aren't needed, it is generally recommended to split them into sub-AnimSets and load them when only needed.

For example, different weapons can have their own AnimSet, unless you need all the weapon animations all the time.

Matinee:

This problem can be worse when it comes to Matinee.

It can be very cumbersome to maintain different AnimSets per Matinee sequence or per level, depending on your workflow (i.e. different person working on different Matinee sequences). Ideally, you would like to have one AnimSet per level, but maintaining this could be very time consuming between different groups of developers (LDs/Animators/Scripters).

The flag, bShouldBakeAndPrune (CL 259515) in InterpData, helps in this situation. It creates new AnimSet for the level and duplicates only the animations that are used by the level and re-links the reference. If the AnimSet is cooked/loaded anyway, then this flag needs to be off because it will duplicate the the same animation. This needs to be used if the AnimSet you're baking and pruning doesn't have to be loaded by the level. The flag can be globally disabled by setting the bBakeAndPruneDuringCook value to FALSE in the Cooker.MatineeOptions section of *Editor.ini. 

In *Editor.ini,
      [Cooker.MatineeOptions]
         bBakeAndPruneDuringCook=false

Animation Compression:

Animation Compression is a big part of saving animation memory. See the Animation Compression page for more information.

Reference System

As the levels from the game get closer to completion, it is well worth taking a pass over all of them to ensure that only the assets needed are loaded. In Unreal Engine 3, any referenced content will be loaded in the game, and with so many ways to reference content – both directly and indirectly – you can end up loading large assets without realizing it. For example, a Pawn referencing a Skeletal Mesh automatically will load the AnimSet that is referenced by the Skeletal Mesh, thus all animations(Pawn->Skeletalmesh->Animsets->Animations). This chain relation can be dangerous in terms of memory as the game gets more Actors referencing each other.

You will normally have to work with a level designer to determine if there are things being loaded that should not be, or that could be split across streaming levels and not all loaded at the same time. An example might be the mesh of a enemy that is not present in this level.

MEMLEAKCHECK:

Memleakcheck prints a lot of basic game data to one output file as text file in [ProfileDirectory:Platform-specific]/MemLeak.

This will at least show you the snapshot of object list of Skeletal Meshes, AnimSets, Sounds, and so on. Once you have a basic snapshot, you can go through them and see if there are unnecessary assets loaded. If so, trace it down to who references it and remove the reference is the key to the solution.

Below is the case study...

OBJ LIST CLASS=SKELETALMESH:

Lists all SkeletalMeshes loaded, sorted by size. MEMLEAKCHECK includes this information, but it is sorted alphabetically, which makes it hard to find content to optimize.

LISTSOUNDS:

Lists all SoundCues loaded, sorted by size. MEMLEAKCHECK includes this information, but it is also sorted alphabetically, which makes it hard to find content to optimize.

OBJ LIST CLASS=STATICMESH: Lists all StaticMeshes loaded, sorted by size.

LISTANIMSETS:

Lists all AnimSets loaded, sorted by size. MEMLEAKCHECK also includes this info.

OBJ REFS:

The most useful way to find why an asset is being loaded is using the OBJ REFS command. To use this, you need to run the game on PC, as it requires a deep stack due to the recursive functions and usually crashes the console versions of UE3.

Play to the point where the asset in question is loaded, and then type: OBJ REFS CLASS= NAME=

For example: 

OBJ REFS CLASS=SKELETALMESH NAME=BIG_OGRE_2

It will then show what the chain of references is that is pulling in this asset e.g.: 

Log: Shortest reachability from root to SkeletalMesh COG_Bolo_Grenade.Frag_Grenade:
Log:    SkeletalMesh COG_Bolo_Grenade.Frag_Grenade [target] (root) (standalone)
Log:    SkeletalMeshComponent GearGame.Default__GearProj_FragGrenade:COGGrenadeMesh0 (root) (ObjectProperty Engine.SkeletalMeshComponent:SkeletalMesh)
Log:    Class GearGame.GearProj_FragGrenade (root) (standalone)

In the example above, the Frag_Grenade mesh is being referenced by the default properties of the GearProj_FragGrenade native class.

Note that there may be more than one reference, in which case you will have to take care of them one at a time.

Here are some reasons we have found for assets being incorrectly loaded:

  • The asset is referenced by a native class.
  • UnrealScript code references the class (that references the asset) to get a default property from it: 
                      Asset = class'MyGameContent.Pawn_BigOgre'.default.Mesh.PhysicsAsset;
  • A Touch Kismet event (SeqEvt_Touch) references the class in its ClassProximityTypes or IgnoredClassProximityTypes arrays.

Memory Fragmentation

MEM STAT: summary only

MEM DETAILED:

This prints out useful information for allocator. Please refer platform section below.

It is a good idea to keep a buffer for memory fragmentation.

Stack Memory

Stack memory is inherited by the main thread. Although you can specify what your stack size is for the thread you create, if you're using library that creates threads internally, it might not consider the memory.

References

Following pages provide a lot more detail on memory information.

Memory On Consoles

Useful Commands