The simulation

Walkthrough the process of creating your first Hadean Simulate simulation

Getting started

To create a Simulate simulation, all you need is a C++ file to get started. In our Hello World, the simulation code is contained in main.cc. Navigate to the location you downloaded HelloWorld.zip to, extract the archive and open main.cc in your code editor to get started.

For this simulation we are going to use Simulate's EnTT integration to store and query our entities and their components.

Data structures

All simulations will require some data to exist for each entity in the simulation. You can give entities data by attaching components to them - components are just C++ structures containing whatever information you need. You can attach as many components as you need to an entity, and they can be as large or as small as you need. Best practice here is to have components that group logically similar functionality together - if you have entities that need to respond to a physics simulation, then creating a physics component is a reasonable way to go!

For our multiplayer game, all entities will need some basic data so we know where they are and how to render them:

struct entity_component {
    aether::geometry::vec3f position;
    uint64_t entity_id;
    float rotation;
    CharacterModelIndex model;
    AnimationIndex animation;
};

As entities may need to be shared between workers within the Simulate simulation, Simulate needs to know how to be able to send the component data over the network. As a result, all components need to be able to be serialised. For trivial types, Simulate provides a macro to handle this for us:

AETHER_SERDE_DERIVE_TRIVIAL(entity_component);

Alongside our data that all entities need - our players also need a player ID, so we can create a second component to hold this:

struct player_component {
    uint64_t player_id;
};

AETHER_SERDE_DERIVE_TRIVIAL(player_component);

Finally several places in the Aether API need the list of all possible components, so we can create a helper tuple type to register them all:

using component_types = std::tuple<entity_component, player_component>

Configuring Hadean Simulate

At its core, Simulate is responsible for managing chunking up the world into smaller cells and assigning a worker to each of those cells. In order to know how to do this, Simulate needs to know how many dimensions to split the space up in. Here you can choose between splitting the space along two dimensions or three.

Choosing two dimensions for Simulate does not limit you to a two dimensional simulation - your entities could still have three dimensional positions. All we are asking for here is for Simulate to consider only two dimensions when deciding how to divide up the space.

If you are unsure whether to use two or three dimensions, consider if you expect your entities to be distributed along the third axis. If your entities mainly exist on a ground plane (e.g. soldiers in a city), then even if they can jump or have a few helicopters above, a two dimensional split may be appropriate. But in a space shooter where players could be anywhere, a three dimensional split may be more appropriate.

To tell Simulate to use two dimensions, we create an octree_traits type based on the aether::vec2i64 type.

using octree_traits = aether::octree_traits<aether::vec2i64>;

Now we can create a simple helper struct that contains all these details for Simulate to use. Here we choose a store_type of aether::entt::store, which enables Simulate's EnTT integration. You can instead choose to use Simulate's built in ECS here. For more details, please see the Entity Component System page. The remainder of this Hello World will assume you are using the EnTT integration.

struct entity_store_traits {
   using octree_traits = octree_traits;
   using handover_type = aether::default_handover;
   using store_type = aether::entt:store;

   template<typename Writer>
   using serializer_type = serializers::agent_serializer<Writer>;

   template<typename Reader>
   using deserializer_type = serializers::agent_deserializer<Reader>;
};

We've excluded the definition of the agent_serializer and agent_deserializer classes here for brevity, they handle the full serialization of an entity rather than for individual components as we've seen previously.

Adding gameplay logic

All our gameplay logic will be executed by the workers, and the main entry-point for them is the user_cell_state_implclass. Here we can override methods to hook into the lifecycle of the worker.

class user_cell_state_impl : public aether::entity_based_user_state_base<entity_store_traits> {
private:
    store_type store;

public:
    user_cell_state_impl(const aether_state_type& aether_state) : store{} {}
    store_type& get_store() override { return store; }

    void initialise_world(const aether_state_type& aether_state) override;
    void initialise_cell(const aether_state_type& aether_state) override;
    void cell_tick(const aether_state_type& aether_state, float delta_time) override;
    vector_type agent_center(const aether_state_type& aether_state, agent_reference agent) override;
    aabb_type agent_aabb(const aether_state_type& aether_state, agent_reference agent) override;
    void serialize_to_client(const aether_state_type& aether_state, client_writer_type& writer) override;
    std::vector<subscriber_topic_type> get_topics(const aether_state_type& aether_state) override;
    void deinitialise_cell(const aether_state_type& aether_state) override;
};

A Simulate simulation has the following lifecycle:

Startup

During startup, intialise_world and initialise_cell provide whole-simulation and per-cell startup hooks. In these you can perform any initialisation you require, either creating entities that should exist from the beginning of the simulation (in initialise_world), or creating state that should exist per-cell. If you are using Simulate's built-in ECS, then you will want to register your systems in initialise_cell so that each cell runs them. For Simulate's EnTT integration this is not required.

Simulate requires that at least one entity exist in the simulation, and so a typical implementation of initialise_world will create a single entity with no components.

void initialise_world(const aether_state_type& aether_state) override {
    // Create a single entity in the world, which will exist for the whole
    // simulation. This uses the EnTT integration.
    store.registry.create();
}

During initialisation, initialise_world will only be called on the first worker that is spawned, this will happen after intialise_cell for that worker, but before initialise_cell on any other workers.

Cell Tick

The cell tick is where most of the core gameplay lives within a simulation. Here you can respond to player input, manipulate the state of entities, and send data to clients to be rendered. The manipulation of entities is handled by systems. When using Simulate's EnTT integration, systems are just functions that are called from within cell_tick.

If you are using Simulate's built-in ECS, then you will register your systems in each worker duringcell_initialiseand incell_tickcallstore.tickto invoke them.

Within cell_tick you can run whatever logic you need in order to update the entity state as appropriate. You could perform a physics simulation, respond to player input, or update AI. In this sense cell_tick is the lowest level hook for building the simulation, and it does not impose any further structure on how you organise your simulations. In order to update entity data Simulate provides access to entities and their components, these can be accessed through the store or registry , depending on if you are using Simulate's built-in ECS or EnTT integration. Regardless of which ECS you are using, Simulate divides entities into two types - local entities and ghost entities.

Local entities

Local entities are entities whose position is inside the cell the current worker is responsible for at the start of cell_tick. These entities the worker has full control over - you can read any data you want and modify it in any way you need to.

Ghost entities

Ghost entities are entities whose position is outside the cell the current worker is responsible for at the start of cell_tick. The current worker does not have full control over these entities - instead these should be treated as a read-only version of the entity. The read-only data will also only be refreshed at the beginning of each tick, with another worker responsible for updating the state of that entity. Ghost entities allow you to respond to entities that are slightly outside of your cell - if you have an AI guard they need to see a player walking in front of them, regardless of being in the same cell or not!

In order to know if a particular entity should have a ghost copy provided to another worker, Simulate requests an AABB for each entity. If this AABB intersects another worker's cell, then a ghost copy of that entity will be made available to that worker. The temptation here is to make these AABB's as large as possible, so that you always have all entities available everywhere, however doing this will make your simulation very slow. Remember that workers may be running on separate machines, and are certainly running in separate processes. As a result, any entities that need to be handed over must be serialised and potentially sent over a network. Doing this for all entities can lead to performance problems, and so you should look to minimise the size of the AABB for each entity, whilst still making them large enough to support your simulation logic.

We will see later a few additional features that Simulate provides to allow some entities to share data over much longer distances.

Once you have updated all the entities during cell_tick, Simulate will need to update the representation it holds of where entities are, and what cells entities should be handed over to. There are two callbacks that you must provide in order to tell Simulate the information it needs.

vector_type agent_center(const aether_state_type& aether_state, agent_reference agent) override {
        if (store.registry.has<entity_component>(agent)) {
            const auto& entity_c = store.registry.get<entity_component>(agent);
            return {
                static_cast<int64_t>(entity_c.position.x),
                static_cast<int64_t>(entity_c.position.z)
            };
        }
        return { 0, 0 };
    }

aabb_type agent_aabb(const aether_state_type& aether_state, agent_reference agent) override {
    const float handover_range = 5.0f;

    if (store.registry.has<entity_component>(agent)) {
        const auto& entity_c = store.registry.get<entity_component>(agent);
        const auto& p = entity_c.position;
        return {
            {
                static_cast<int64_t>(p.x - handover_range),
                static_cast<int64_t>(p.z - handover_range)
            },
            {
                static_cast<int64_t>(p.x + handover_range),
                static_cast<int64_t>(p.z + handover_range)
            },
        };
    }

    return { { 0, 0 }, { 0, 0 } };
}

agent_center is the callback Simulate provides so you can report the position of entities to Simulate. This position must be in integer coordinates - typically this is done by flooring the components of the position. This position should be calculated only from the entity itself, and not require looking at other entities. As a result if you need to position an entity relative to another entity, you should update the position of each entity accounting for any relative offset in your cell_tick method so that you have the position fully available in this callback.

agent_aabb is the callback Simulate provides so you can describe how far away entities should be able to see the current entity. Again the vectors describing the AABB should be floored to integer coordinates. The AABB should also enclose the position described by agent_center for the entity.

Entities, components and systems

In a Simulate simulation, individual agents are modelled as entities. Each entity can contain a set of components that model the state of that entity. These components can contain as much or as little data as required for your simulations, although typically the preference is for more granular components, which can be composed together to create more complex state. Components do not themselves tend to have and logic (although Simulate does not strictly impose this restriction), instead components are acted upon by systems, and these systems contain the logic for updating the state of entities by interacting with their components.

Simulate imposes a singular requirement on components in order to use them - it must be possible to serialise and deserialise them. This is required so that as entities move between workers, the data can be transferred. It also powers the ghost entities, ensuring that an up-to-date copy of entity state can be maintained on workers for nearby cells.

For a complete description of serialising and deserialising components, please see Entity Component System. For any trivially copyable types, you can use the macro AETHER_SERDE_DERIVE_TRIVIAL in order to implement serialisation automatically.

When using the EnTT integration, you have full access to the EnTT API to query for entities and look up components. Systems to interact with components are just functions, and you can call these functions directly from your cell_tick method.

void cell_tick(const aether_state_type& aether_state, float delta_time) override {
    incoming_messages_system(store.registry, &incoming_message_buffer, &outgoing_message_buffer);
}

Here we pass store.registry, which is the EnTT component registry, so the system can act upon our entities.

void incoming_messages_system(aether::entt::registry_type& registry,
    aether::message::message_buffer* incoming_messages,
    aether::message::message_buffer* outgoing_messages) {

    auto reader = incoming_messages->get_message_reader();
    while (auto maybe_message = reader.get_next()) {
        const auto& message = maybe_message.value();
        const auto maybe_player_id = message.get_source_user_id();

        protocol::event_connect connect_event;
        if (message.payload_as_pod(connect_event)) {

            auto player_entity = registry.create();

            auto& player_c = registry.emplace<player_component>(player_entity);
            player_c.player_id = connect_event.player_id;

            auto& entity_c = registry.emplace<entity_component>(player_entity);
            entity_c.entity_id = connect_event.entity_id;
            entity_c.position = connect_event.position;
            entity_c.rotation = connect_event.rotation;
            entity_c.model = connect_event.model;
            entity_c.animation = connect_event.animation;

            continue;
        }
    }
}

In this system we are responding to incoming messages (this will be explored in more detail below). To create a new entity in our world we call registry.create(), which gives us a handle to the created entity. We can then create components and attach them to this entity with registry.emplace<component>(entity)which returns a reference to the created component. This reference can be mutated directly to update the state of the component.

In order to find entities that have a specific combination of components, we can use EnTT's views. This returns an object that can act as an iterator, or we can pass a callback to the .each() method.

registry.view<const player_component, entity_component>()
    .each([&](auto e, const auto& player_c, auto& entity_c) {
        // Transform your components here
    });

As you can see, the .each() callback is given a handle to the current entity, plus a reference to each of the requested components.

Ghost entities

When using views to query for entities that have a component, Simulate will include any ghost entities available to the current worker in the result of this query. This allows you to respond to nearby entities (for example, to do collision avoidance), without worrying about being near a cell boundary. However these ghost entities are replicas of the entity, and any mutations you make to them will not be persisted to the next tick. Instead a fresh copy of the state of that entity will be transferred from the worker that owns the entity each tick.

You can identify entities that are ghosts as they will have an aether::entt::component::ghost tag component on them. If you want to create a view that only considers local entities, then you can use the exclusion filters on EnTT views in order to do so:

registry.view<const player_component, entity_component>(
    entt::exclude<aether::entt::component::ghost>
).each([&](auto e, const auto& player_c, auto& entity_c) {
    // Transform your components here
});

Receiving messages

Using systems to operate on entities allows some complex scenarios to be created. However frequently we want to be able to send data from an external system to our simulation, or send a notification from a worker to another worker. In order to support this, Simulate provides a messaging system, which supports the sending and receiving of abitrary messages.

We have already seen an example of receiving messages, where in our incoming_messages_system we used a message reader to read any incoming messages. The message reader allows us to read through messages with the get_next() method. The message returned is an optional value, which will be populated when a message is available. Messages are simple byte arrays, and it is up to you to decide how to identify the type of message. Messages are also decorated with information on the source of the message.

Looking again at the incoming messaging system, we can see this in action.

void incoming_messages_system(aether::entt::registry_type& registry,
    aether::message::message_buffer* incoming_messages,
    aether::message::message_buffer* outgoing_messages) {

    auto reader = incoming_messages->get_message_reader();
    while (auto maybe_message = reader.get_next()) {
        const auto& message = maybe_message.value();
        const auto maybe_player_id = message.get_source_user_id();

        protocol::event_connect connect_event;
        if (message.payload_as_pod(connect_event)) {
            // Snip: Create entity as before
            continue;
        }

        protocol::event_move move_event;
        if (message.payload_as_pod(move_event) && maybe_player_id.has_value()) {
            // Snip: Update the player position in response to move_event
            continue;
        }
    }
}

Here we can see the creation of the message reader, and using get_next() to read any availabe messages. get_source_user_id() allows us to discover if the message has been sent by a player. If the value returned holds an ID, this will be the ID of the connected user sending the message. If this value does not hold an ID, then this means the message was sent from something internal to Simulate, either another worker or the background worker.

In this case we are using the payload_as_pod method to disambiguate the type of the message. This works by considering the size of the message, if it is the same as the size of the type being checked, then the message is assumed to be of this type. As our protocol::event_connect and protocol::event_move types have different sizes, we can use this to disambiguate them.

When Simulate is processing a message, it needs to know which workers should receive it. Consider a message being recieved from a player - sending this to all workers would be wasteful, given only a single worker will have the entity that the player is controlling. In order to let Simulate know which workers the messages should be routed to, the get_topics() method will be called that allows you to populate a list of message topics. Normally this will include messages for any players that the worker owns the entity the player is controlling.

virtual std::vector<subscriber_topic_type> get_topics(const aether_state_type& aether_state) override {
    std::vector<subscriber_topic_type> topics;

    store.registry.view<const player_component>(entt::exclude<aether::entt::component::ghost>)
        .each([&](auto e, const auto& player_c) {
            topics.push_back(aether::message::topic::user_id{ player_c.player_id });
        });

    return topics;
}

You can also subscribe to topics for individual entities, or custom destinations.

Background processing

When creating simualtions, you may come across scenarios in which it is necessary to perform some calculation, but where that calculation does not map directly to a portion of the space or a specific entity. You may also find that you need to hold some global state that should persist irrespective of what is happening with individual workers. To facilitate this, Simulate provides a background worker that you can use for these tasks.

Communication with the background worker is done exclusively through messaging - it does not have access to any entities directly. As with other workers you can subscribe to topics to receive messages, and the process_messages callback will be used to allow you to respond to messages.

class background_worker : public aether::global_state_base<octree_traits> {
public:
    void process_messages(reader_type& reader, writer_type& writer) override {
        while (auto maybe_message = reader.get_next()) {
            const auto& message = maybe_message.value();
            const auto maybe_player_id = message.get_source_user_id();

            protocol::event_connect connect_event;
            if (maybe_player_id.has_value() && message.payload_as_pod(connect_event)) {
                const aether::vec2i64 target_position = {
                    static_cast<int64_t>(connect_event.position.x),
                    static_cast<int64_t>(connect_event.position.z),
                };
                writer.set_destination(
                    aether::message::closest_worker{ target_position }
                );

                connect_event.entity_id = next_id++;

                writer.push_bytes(&connect_event, sizeof(connect_event));
                writer.send();
            }
        }
    }

    std::vector<subscriber_topic_type> get_topics() override {
        std::vector<subscriber_topic_type> topics;
        topics.push_back(aether::message::topic::unclaimed_events{});
        return topics;
    }

private:
    uint64_t next_id = 0;
};

Here our background worker subscribes to any unclaimed events. This is a fallback topic that is used when a message is received but does not match the subscriptions of any other workers. One of the main use cases of this is when a player first connects to a simulation, no workers will have an entity representing the player. As a result, none will be subscribed to receive messages from that player, so the background worker will receive the message. In this case we respond by assigning an entity ID to the player's entity, and then forwarding the message to the closest worker to where the player is spawning.

Sending data to clients

In most simulations, simply changing the state of the entities in your simulation will not be sufficient without being able to see the current state of those entities. In order to do this, we must serialise the state of all the entities that we wish to expose from our simulation to any connected clients. When using Simulate this is done by first sending data to the Connect. You can send any arbitrary byte stream that you want to and write a custom netcode to handle that data and redistribute it as appropriate to individual clients. As part of your SDK installation however, a simple generic netcode is provided, which uses a set of marshalling classes to easily add entity data.

The data that is sent out of an application is often one of the main bottlenecks in the application. You should expect to spend time tuning the data being sent out from your simulation to clients to find the right balance between the fidelity of the data and the data volume. In the next pages we will look at Connect, and the closely related netcode to see how some of the problems associated with these data volumes are solved, these approaches are based on reducing the number of entities we send data for. However for each entity there are several techniques that can be employed to help reduce the data bandwidth required:

  • Only send the data the clients need. Although obvious, not sending data is always the fastest thing to do, and so if your clients do not need some information to present, do not send it to them.

  • Reduce the precision of the data. Often entity data does not need to be as precise to be rendered on the client as it does in the simulation. Consider if any double fields can be sent as float fields, or if a uint64_t is really required rather than a uint32_t or uint16_t. You may be able to extend this if you know more about your simulation. For example, if your simulation only supports entities that have a yaw rotation rather than pitch and roll, then you do not need a full quaternion to send this data to clients, and can instead use a single float.

  • Compress your data. Often running data through a compression library can reduce data transfer sizes significantly, even for binary data.

  • Quantise values. Often for floating point values the full float precision is not required, and quantising the values is enough. For example, rotations can be expressed in a single byte by only allowing 256 possible rotation angles around an axis. This is may be sufficient for rendering, and saves 3 bytes on using a full float. We have seen examples where combining this with swapping quaternions for floats representing yaw rotations saves a full 15 bytes for the rotation (~93%).

  • Combine multiple values into a single byte. For integers that only have a small range (e.g. 0-31, or -16-15), multiple values can be combined into a single byte rather than having to sending a byte or more for each value. This can work well for flags indicating the state or enumerations with a limited set of possible values.

There are many such techniques to consider when sending these values to a client. Generally speaking the more specific you can be about the data you send, the more scope there is for optimisation.

In order to send data to the Connect, Simulate will after each tick invoke the serialize_to_clientmethod. In this method, you can use the writer argument to send an arbitrary byte stream to Connect. Indeed if you so wished you could use a fully standardised protocol, such as flatbuffers or protobuf, or indeed serialise to a human-readable format such as JSON. Normally you will want to serialise the data to a binary format to reduce the data size.

As the data sent between the simulation, netcode and client must always follow the same protocol in order to be understood, we move this information to a separate header file.

protocol.hh
namespace protocol {
    HADEAN_PACK(struct per_entity_data {
         aether::geometry::vec3f position;
         uint64_t playerId;
         uint64_t entityId;
         float rotation;
         CharacterModelIndex model;
         AnimationIndex animation;
         uint8_t state;   
    });
}

struct trivial_marshalling_traits {
     using per_worker_data_type = aether::protocol::base::client_message;
     using entity_type = protocol::per_entity_data;
     using static_data_type = std::monostate;
};
using marshalling_factory = aether::netcode::trivial_marshalling<trivial_marshalling_traits>;

Now that the marshaller is available, we can start to use it in our serialize_to_client method.

main.cc
void serialize_to_client(const aether_state_type& aether_state, client_writer_type& writer) override {
    using namespace aether::protocol::base;

    auto marshaller = marshalling_factory().create_marshaller();

    client_message header = {
        .cell = {
            aether::to_machine_id(hadean::get_pid()),
            aether_state.get_cell().level,
            aether_state.get_cell().position
        },
        .cell_dying = aether_state.is_cell_dying(),
        .stats = {
            .num_agents = store.num_agents_local(),
            .num_agents_ghost = store.num_agents_ghost()
        },
    };

    marshaller.add_worker_data(aether_state.get_worker().as_u64(), header);

    store.registry.view<entity_component, player_component>(entt::exclude<aether::entt::component::ghost>)
        .each([&](auto e, auto& entity_c, auto& player_c) {
            protocol::per_entity_data entity = {
                .position = entity_c.position,
                .playerId = player_c.player_id,
                .entityId = entity_c.entity_id,
                .rotation = entity_c.rotation,
                .model = entity_c.model,
                .animation = entity_c.animation
            };
            marshaller.add_entity(entity);
        });

    const auto data = marshaller.encode();

    writer.push_bytes(data.data(), data.size());
    writer.send();
}

Here we create an instance of the marshaller, and use add_worker_data to append a header describing the current state of the cell. This is not strictly necessary, but can be used by the client to render information on the layout of cells in the Simulate simulation.

We then use an EnTT view to find all of our entities excluding any ghosts, and append each one to the message to be sent to the Connect. Finally we use the marshaller to encode this into a byte stream that is sent to the Connect.

It is important to exclude ghosts when serialising data to be sent to the client, as without this clients may receive duplicate values for entities.

Here a more complex simulation will likely use multiple views to append entity data, transforming it as needed.

Now that we have an Simulate simulation that allows us to send and receive data from clients, we can look at how Connect handles transferring this state to thousands of connected players.

Last updated