FNetFastTArrayBaseState

===================== NetSerialize and NetDeltaSerialize customization. =====================

The main purpose of this file it to hold custom methods for NetSerialization and NetDeltaSerialization. A longer explanation on how this all works is covered below. For quick reference however, this is how to customize net serialization for structs.

To define your own NetSerialize and NetDeltaSerialize on a structure: (of course you don't need to define both! Usually you only want to define one, but for brevity Im showing both at once) ===================== Fast TArray Replication =====================

Fast TArray Replication is a custom implementation of NetDeltaSerialize that is suitable for TArrays of UStructs. It offers performance improvements for large data sets, it serializes removals from anywhere in the array optimally, and allows events to be called on clients for adds and removals. The downside is that you will need to have game code mark items in the array as dirty, and well as the order of the list is not guaranteed to be identical between client and server in all cases.

Using FTR is more complicated, but this is the code you need:Step 6 and beyond: -Declare a UPROPERTY of your FExampleArray (step 2) type. -You MUST call MarkItemDirty on the FExampleArray when you change an item in the array. You pass in a reference to the item you dirtied. See FFastArraySerializer::MarkItemDirty. -You MUST call MarkArrayDirty on the FExampleArray if you remove something from the array. -In your classes GetLifetimeReplicatedProps, use DOREPLIFETIME(YourClass, YourArrayStructPropertyName);

You can override the following virtual functions in your structure (step 1) to get notifies before add/deletes/removes: -void PreReplicatedRemove(const FFastArraySerializer& Serializer) -void PostReplicatedAdd(const FFastArraySerializer& Serializer) -void PostReplicatedChange(const FFastArraySerializer& Serializer)

Thats it!

===================== An Overview of Net Serialization and how this all works ===================== Everything originates in UNetDriver::ServerReplicateActors. Actors are chosen to replicate, create actor channels, and UActorChannel::ReplicateActor is called. ReplicateActor is ultimately responsible for deciding what properties have changed, and constructing an FOutBunch to send to clients.

The UActorChannel has 2 ways to decide what properties need to be sent. The traditional way, which is a flat TArray<uint8> buffer: UActorChannel::Recent. This represents a flat block of the actor properties. This block literally can be cast to an AActor* and property values can be looked up if you know the FProperty offset. The Recent buffer represents the values that the client using this actor channel has. We use recent to compare to current, and decide what to send.

This works great for 'atomic' properties; ints, floats, object*, etc. It does not work for 'dynamic' properties such as TArrays, which store values Num/Max but also a pointer to their array data, The array data has no where to fit in the flat Recent buffer. (Dynamic is probably a bad name for these properties)

To get around this, UActorChannel also has a TMap for 'dynamic' state. UActorChannel::RecentDynamicState. This map allows us to look up a 'base state' for a property given a property's RepIndex.

NetSerialize & NetDeltaSerialize Properties that fit into the flat Recent buffer can be serialized entirely with NetSerialize. NetSerialize just reads or writes to an FArchive. Since the replication can just look at the Recent[] buffer and do a direct comparison, it can tell what properties are dirty. NetSerialize just reads or writes.

Dynamic properties can only be serialized with NetDeltaSerialize. NetDeltaSerialize is serialization from a given base state, and produces both a 'delta' state (which gets sent to the client) and a 'full' state (which is saved to be used as the base state in future delta serializes). NetDeltaSerialize essentially does the diffing as well as the serialization. It must do the diffing so it can know what parts of the property it must send.

Base States and dynamic properties replication. As far as the replication system / UActorChannel is concerned, a base state can be anything. The base state only deals with INetDeltaBaseState*.

UActorChannel::ReplicateActor will ultimately decide whether to call FProperty::NetSerializeItem or FProperty::NetDeltaSerializeItem.

As mentioned above NetDeltaSerialize takes in an extra base state and produces a diff state and a full state. The full state produced is used as the base state for future delta serialization. NetDeltaSerialize uses the base state and the current values of the actor to determine what parts it needs to send.

The INetDeltaBaseStates are created within the NetDeltaSerialize functions. The replication system / UActorChannel does not know about the details.

Right now, there are 2 forms of delta serialization: Generic Replication and Fast Array Replication.

Generic Delta Replication Generic Delta Replication is implemented by FStructProperty::NetDeltaSerializeItem, FArrayProperty::NetDeltaSerializeItem, FProperty::NetDeltaSerializeItem. It works by first NetSerializing the current state of the object (the 'full' state) and using memcmp to compare it to previous base state. FProperty is what actually implements the comparison, writing the current state to the diff state if it has changed, and always writing to the full state otherwise. The FStructProperty and FArrayProperty functions work by iterating their fields or array elements and calling the FProperty function, while also embedding meta data.

For example FArrayProperty basically writes: "Array has X elements now" -> "Here is element Y" -> Output from FProperty::NetDeltaSerialize -> "Here is element Z" -> etc

Generic Data Replication is the 'default' way of handling FArrayProperty and FStructProperty serialization. This will work for any array or struct with any sub properties as long as those properties can NetSerialize.

Custom Net Delta Serialiation Custom Net Delta Serialiation works by using the struct trait system. If a struct has the WithNetDeltaSerializer trait, then its native NetDeltaSerialize function will be called instead of going through the Generic Delta Replication code path in FStructProperty::NetDeltaSerializeItem.

Fast TArray Replication Fast TArray Replication is implemented through custom net delta serialization. Instead of a flat TArray buffer to repesent states, it only is concerned with a TMap of IDs and ReplicationKeys. The IDs map to items in the array, which all have a ReplicationID field defined in FFastArraySerializerItem. FFastArraySerializerItem also has a ReplicationKey field. When items are marked dirty with MarkItemDirty, they are given a new ReplicationKey, and assigned a new ReplicationID if they don't have one.

FastArrayDeltaSerialize (defined below) During server serialization (writing), we compare the old base state (e.g, the old ID<->Key map) with the current state of the array. If items are missing we write them out as deletes in the bunch. If they are new or changed, they are written out as changed along with their state, serialized via a NetSerialize call.

For example, what actually is written may look like: "Array has X changed elements, Y deleted elements" -> "element A changed" -> Output from NetSerialize on rest of the struct item -> "Element B was deleted" -> etc

Note that the ReplicationID is replicated and in sync between client and server. The indices are not.

During client serialization (reading), the client reads in the number of changed and number of deleted elements. It also builds a mapping of ReplicationID -> local index of the current array. As it deserializes IDs, it looks up the element and then does what it needs to (create if necessary, serialize in the current state, or delete).

There is currently no delta serialization done on the inner structures. If a ReplicationKey changes, the entire item is serialized. If we had use cases where we needed it, we could delta serialization on the inner dynamic properties. This could be done with more struct customization.

ReplicationID and ReplicationKeys are set by the MarkItemDirty function on FFastArraySerializer. These are just int32s that are assigned in order as things change. There is nothing special about them other than being unique.Custom INetDeltaBaseState used by Fast Array Serialization