Netcode

Documentation Objective

Explains how to implement the muxer netcode interface.

Overview

This document describes how to implement the muxer netcode interface from scratch. It is recommended using the provided Generic Netcode library first, and only implement this interface for additional customisation that isn’t possible with Generic Netcode.

Implementing the muxer netcode interface

The netcode interface consists of the following functions (see aether/muxer/netcode_api.hh):

void *new_netcode_context();
void destroy_netcode_context(void *);
void netcode_new_simulation_message(void *ctx, void *muxer, uint64_t worker_id, uint64_t tick, const void *data, size_t data_len);
void netcode_new_connection(void *ctx, void *muxer, void *connection, uint64_t id);
void netcode_drop_connection(void *ctx, void *muxer, uint64_t id);
void netcode_notify_writable(void *ctx, void *muxer, uint64_t id);

Netcode constructor and destructor

void *new_netcode_context()

This function produces a context which will be passed to all other netcode related functions. This context can be used to maintain whatever state is needed in order to implement net relevancy. Typically this will include maintaining per-client state on what each client has seen as well as some representation of the world state. Since the returned type is a void*, it will need to be casted back to the appropriate type in all other functions.

void destroy_netcode_context(void *ctx)

When the muxer has finished with a context, it will use this function to destroy it.

Connection Handling

void netcode_new_connection(void *ctx, void *muxer, void *connection, uint64_t id)

This function is called when a new connection has been made to the muxer.

  • ctx is the netcode context that was created with net_netcode_context()
  • muxer is a pointer to a muxer context which can be used in other function calls.
  • connection is a pointer to a connection context which can be used in other function calls.
  • id is an integer identifier assigned to this connection by the muxer.

A typical netcode implementation will update the netcode context to track state for the new connection.

void netcode_drop_connection(void *ctx, void *muxer, uint64_t id)

This function is called when a connection to the muxer is dropped.

  • ctx is the netcode context that was created with net_netcode_context()
  • muxer is a pointer to a muxer context which can be used in other function calls.
  • id is an integer identifier that was previously assigned to this connection.

A typical netcode implementation will update its state to remove tracking information for this connection. It is no longer valid to use the connection state that was associated with the connection id when netcode_new_connection() was called.

The function aether::netcode::release_connection() must be called with the original connection context supplied by netcode_new_connection() during the execution of this function. This requirement will likely disappear in a later version of the SDK.

Message handling

void netcode_new_simulation_message(
void *ctx, void *muxer, uint64_t worker_id, uint64_t tick,
const void *data, size_t data_len)

This function is called when the muxer receives a new message from the simulation.

  • ctx is the netcode context that was created with net_netcode_context()
  • muxer is a pointer to a muxer context which can be used in other function calls.
  • worker_id is a unique identifier for a particular worker. It should not be assumed that this numbering corresponds to the worker IDs used inside simulation code.
  • tick is the tick number this message was produced at.
  • data is the binary content of the message received.
  • data_len is the length of the binary content in bytes.

A typical netcode implementation may choose to decode the binary payload into simulation entities then update the world state with new information. Since there is no guarantee that messages are received in monotonically increasing order, tick can be used determine whether this message contains the most up-to-date information for a given entity.

This function does not typically queue data to be sent to clients directly. Instead this is done in the netcode_notify_writable() function which is called when data can be written to the client. When there is no data to write to a client, a typical implementation will unsubscribe from writable notifications for that client. If the new message contains information that will cause more data to be written to a client that has been unsubscribed from writable notifications, they should be re-enabled with aether::netcode::connection_subscribe_writable().

void netcode_notify_writable(void *ctx, void *muxer, uint64_t id)

This function is called whenever a connection to a client is able to accept more data from the buffer.

  • ctx is the netcode context that was created with net_netcode_context()
  • muxer is a pointer to a muxer context which can be used in other function calls.
  • id is an integer identifier that was previously assigned to this connection.

This function must call aether::netcode::connection_notify_writable(void *connection_ctx, void *muxer_ctx) as its first action with the muxer context, and connection context associated with id. This requirement will likely disappear in a later version of the SDK.

It is then possible to check if the previous message has been completely written to the client using aether::netcode::connection_is_drained(). If this is the case, then a new message can be written to the client using aether::netcode::connection_push_packet().

If writing new message to this client is not desired, then aether::netcode::connection_subscribe_writable() should be used to unsubscribe from writable notifications, otherwise netcode_notify_writable() will be called continuously for this connection. Notifications for this client can be re-enabled in netcode_new_simulation_message() when a message is received which means more data should be sent.

See the Message Handling API section for more details and the walkthrough for an example.

Muxer netcode API

Muxer provides following functions in aether::netcode namespace (see aether/muxer/netcode_api.hh):

/* returns player id */
uint64_t connection_get_player_id(void *connection_context);
/* see the Connection Handling section above */
void release_connection(void *connection_context);
/* see the Message Handling and Message Handling API sections*/
void connection_push_packet(void *connection_context, void *muxer_context, uint64_t worker_id, const void *data, size_t data_len);
bool connection_is_drained(void *connection_context);
void connection_subscribe_writable(void *connection, void *muxer_context, bool enabled);
void connection_notify_writable(void *connection_context, void *muxer);

Message Handling API

This section expands on some of the description of functions mentioned in the Message Handling section.

void connection_push_packet(void *connection_ctx, void *muxer_ctx, uint64_t worker_id, const void *data, size_t data_len)

Writes a packet to a specified connection. Typically this should only be done if connection_is_drained() returns true for this connection, otherwise the send buffer may fill indefinitely causing the muxer to run out of memory.

  • connection_ctx is the context of the connection to write to.
  • muxer_ctx is the muxer context of the connection.
  • worker_id specifies which worker id the repclient library will identify this packet as coming from. This exists due to backwards compatibility reasons and should be set to zero. It will likely be removed in a later version of the SDK.
  • data is a pointer to the message data to be written to the client. data_len is the length of the message in bytes.
void connection_notify_writable(void *connection_ctx, void *muxer_ctx)

This function should be called from netcode_notify_writable() to drain the data remaining in the connection buffer and push it to the client. Will only drain as much data as can be accepted by the client when called. This function will likely disappear in a future SDK version.

  • connection_ctx is the context of the connection to write to.
  • muxer_ctx is the muxer context of the connection.
bool connection_is_drained(void *connection_context)

Returns true if the buffer of data to be written to a specified connection is empty. When the write buffer of a connection is empty, more data should either be written using connection_push_packet() or writable notifications for the connection should be disabled using connection_subscribe_writable() if the connection is still subscribed to them.

void connection_subscribe_writable(void *connection, void *muxer_context, bool enabled)

This function determines whether or not netcode_notify_writable() will be called for the specified connection when the client connection can accept more data from the connection buffer.

  • connection_ctx is the connection context.
  • muxer_ctx is the muxer context.
  • enabled sets whether callbacks are enabled.

Prerequisites - Aether Hunt example

The walkthrough below is based on theAether Hunt demo , so having access to the full source code of the project is recommended.


First steps - high level overview of the Aether Hunt muxer

  • muxer/muxer.cc contains the implementation of the muxer netcode interface. This implementation simply forwards the method calls to a c++ struct
  • muxer/netcode.hh and muxer/netcode.cc defines struct netcode {...} which implements the netcode logic
  • protocol/aether_hunt_protocol.hh contains protocol definitions shared between the client, the simulation and the muxer (usually struct layouts and constants)
  • muxer/nanoflann.hppis a library used for indexing entities based on their location. It isn’t a part of aether and it isn’t covered in this guide

Walkthrough

1. Register the netcode interface methods and forward the method calls to our netcode struct

In the case of Aether Hunt the method calls are forwarded to a custom C++ struct with all the implementation.

// muxer.cc
extern "C" {
/* 1. use a C++ object as the context */
void *new_netcode_context() {
    return new netcode();
}
void destroy_netcode_context(void *ctx) {
    delete static_cast<netcode*>(ctx);
}
/* 2. cast the context to the right type and simply forward the method call */
void netcode_new_simulation_message(void* ctx, void* muxer, uint64_t worker_id, uint64_t tick, const void* data, size_t data_len) {
    auto nc = static_cast<netcode*>(ctx);
    nc->new_simulation_message(muxer, worker_id, tick, data, data_len);
}
/* 3. do the same for all other methods of the interface */
void netcode_notify_writable(void* ctx, void* muxer, uint64_t id) {
    auto nc = static_cast<netcode*>(ctx);
    nc->notify_writable(muxer, id);
}
/* ... other methods omited for brevity */
}
/* declare the main method of this executable, which calls the muxer library */
extern "C" void muxer_main(ssize_t argc, const char* const* argv);
int main(int argc, const char* const* argv) {
    muxer_main(argc, argv);
}

2. Define the protocol (definitions shared between client, simulation and the muxer)

The structs defined here are wrapped in HADEAN_PACK macro, which specifies the alignment for struct fields, so that there’s less padding sent over the network. The structs here must be memcpy-iable to be properly serialized

#include <aether/common/vector.hh>
#include <aether/common/morton/encoding.hh>
#include <aether/common/packing.hh>
#include <aether/common/base_protocol.hh>
#include <cstdint>
namespace AetherHuntProtocol
{
    // sent only in sim -> muxer comms
// describes each worker/cell
    HADEAN_PACK(struct simulation_header {
        uint64_t tick;
        uint64_t cell_id;
        bool cell_dying;
        protocol::base::net_tree_cell cell;
        uint64_t number_of_predators;
        uint64_t number_of_prey;
        uint64_t number_of_players;
    });
    // sent only in sim -> muxer comms
    HADEAN_PACK(struct player {
        uint64_t player_id;
        uint64_t entity_id;
        bool always_relevant = false;
        bool show_cells = false;
    });
    // sent only in muxer -> user comms
    HADEAN_PACK(struct muxer_header {
        uint64_t player_entity_id;
        uint64_t number_of_predators;
        uint64_t number_of_prey;
        uint64_t number_of_cells;
    });
    HADEAN_PACK(struct predator {
        uint64_t entity_id;
        vec2f position;
        bool is_sprinting;
    });
    HADEAN_PACK(struct prey {
        uint64_t entity_id;
        vec2f position;
        uint8_t flags;
        // flags:
        // 0 0 0 0 0 0 0 1 -> dead
    });
    typedef enum {
        T_PREY = 0,
        T_PREDATOR = 1,
        T_PLAYER = 2,
    } aether_entity_t;
// struct containing the entity update
// example of a discriminated union - saves space if only one object is sent at a time
    HADEAN_PACK(struct simulation_message {
        uint8_t type;
        union {
            player player;
            predator predator;
            prey prey;
        };
    });
    /* ... the rest of the file is omited for brievity */
}

3. Implement the netcode logic

struct netcode {
public:
/* Initialise the spatial indexes used for looking up entities based on
the distance from particular player. */
netcode() :
kd_predators(2, predators_cloud, KDTreeSingleIndexAdaptorParams(10)),
kd_preys(2, preys_cloud, KDTreeSingleIndexAdaptorParams(10)) {
kd_predators.buildIndex();
kd_preys.buildIndex();
}
/* We have a new connection from a client - cache it for later use */
void new_connection(void* muxer, void* connection, uint64_t id) {
assert(connection_pool.find(id) == connection_pool.end());
connection_pool[id] = connection;
}
/* A client has disconnected - remove the connection from the pool and
confirm the removal by calling aether::netcode::release_connection */
void drop_connection(void* muxer, uint64_t id) {
const auto it = connection_pool.find(id);
if(it != connection_pool.end()) {
aether::netcode::release_connection(it->second);
connection_pool.erase(it);
}
assert (false); /* this should never happen */
}
/* New data has been sent from a simulation worker */
void new_simulation_message(void* muxer, uint64_t worker_id, uint64_t tick, const void* data, size_t data_len) {
if(data == nullptr || data_len == 0) {
return;
}
if(header->tick > tick_got) {
tick_got = header->tick;
}
/* update the cached state of all the received entities */
update_cell(header->cell_id, header->cell, header->cell_dying);
update_players(player_list, header->number_of_players);
update_predators(predator_list, header->number_of_predators);
update_prey(prey_list, header->number_of_prey);
/* register for notifications for all clients as
it's possible that all of them require an update */
notify_players(muxer);
}
/* A client connection socket is ready to accept new data */
void notify_writable(void* muxer, uint64_t id) {
const auto it = connection_pool.find(id);
if(it == connection_pool.end()) {
fprintf(stderr, "Received notify_writable for dead connection!\n");
/* this should never happen */
return;
}
auto connection = it->second;
/* ask the muxer to drain the buffered data to the connection socket */
aether::netcode::connection_notify_writable(connection, muxer);
if (!aether::netcode::connection_is_drained(connection)) {
/* the connection buffer hasn't been fully drained yet */
/* return in order to avoid overflowing/reallocating the connection buffer
by pushing more data into it,
we will retry once this callback is called again */
return;
}
const bool state_was_sent = send_simulation_state(muxer, connection);
/* unregister from the writable notification if we didn't push
anything to the connection buffer (state_was_sent == false)*/
/* keep being registered if we pushed to the buffer,
so that we flush all the pushed data (state_was_sent == true)*/
aether::netcode::connection_subscribe_writable(connection, muxer, state_was_sent);
}
private:
/* new data came in,
make sure we receive a callback once clients are ready for receiving it */
void notify_players(void* muxer) {
for(const auto& it : connection_pool) {
aether::netcode::connection_subscribe_writable(it.second, muxer, true);
}
}
/* create the update and push the packet to the connection buffer */
bool send_simulation_state(void* muxer, void* connection) {
const auto player_id = aether::netcode::connection_get_player_id(connection);
const auto it = player_to_entity_map.find(player_id);
if (it == player_to_entity_map.end()) {
return false;
}
/* check if a tick was already sent */
if (!should_send(player_id)) {
return false;
}
/* check if we have the player in cache */
const auto player_it = players.find(player_id);
if (player_it == players.end()) {
return false;
}
AetherHuntProtocol::muxer_header header{};
header.player_entity_id = it->second;
float search_radius = 0;
if(player_it->second.always_relevant) {
/* debug client - send all entity updates */
auto prey_it, predator_it, cell_it;
std::vector<AetherHuntProtocol::predator> relevant_predators;
std::vector<AetherHuntProtocol::prey> relevant_preys;
std::vector<protocol::base::net_tree_cell> relevant_cells;
for ( prey_it = preys.begin(); prey_it != preys.end(); prey_it++ ) {
relevant_preys.push_back(prey_it->second);
}
for ( predator_it = predators.begin(); predator_it != predators.end(); predator_it++ ) {
relevant_predators.push_back(predator_it->second);
}
header.number_of_predators = relevant_predators.size();
header.number_of_prey = relevant_preys.size();
if(player_it->second.show_cells) {
header.number_of_cells = cells.size();
for (cell_it = cells.begin(); cell_it != cells.end(); cell_it++) {
relevant_cells.push_back(cell_it->second);
}
}
else {
header.number_of_cells = 0;
}
/* now that headers and data is ready, send the packet */
build_and_send_packet(header, relevant_predators, relevant_preys, relevant_cells);
} else {
search_radius = NET_RELEVANCY_RADIUS;
/* regular client - only send updates for entities in given radius */
std::vector<std::pair<size_t, float>> ret_predator_matches;
std::vector<std::pair<size_t, float>> ret_prey_matches;
std::unordered_map<size_t, protocol::base::net_tree_cell>::iterator cell_it;
/* search for entities in the relevancy radius */
nanoflann::SearchParams params;
params.sorted = false;
vec2f player_pos = predators[header.player_entity_id].position;
float pos[2]{ player_pos.x, player_pos.y };
header.number_of_predators = kd_predators.radiusSearch(&pos[0], search_radius, ret_predator_matches, params);
header.number_of_prey = kd_preys.radiusSearch(&pos[0], search_radius, ret_prey_matches, params);
std::vector<AetherHuntProtocol::predator> relevant_predators;
std::vector<AetherHuntProtocol::prey> relevant_preys;
std::vector<protocol::base::net_tree_cell> relevant_cells;
for (auto &pair : ret_predator_matches) {
relevant_predators.push_back(predators[predators_idx_to_entity[pair.first]]);
}
for (auto &pair : ret_prey_matches) {
relevant_preys.push_back(preys[preys_idx_to_entity[pair.first]]);
}
if(player_it->second.show_cells) {
header.number_of_cells = cells.size();
for (cell_it = cells.begin(); cell_it != cells.end(); cell_it++) {
relevant_cells.push_back(cell_it->second);
}
}
else {
header.number_of_cells = 0;
}
/* now that headers and data is ready, send the packet */
build_and_send_packet(header, relevant_predators, relevant_preys, relevant_cells);
}
return true;
}
/* we send only one update per tick,
so we need to check what was the last tick sent and update if needed */
bool should_send(uint64_t player_id) {
//
const auto it = tick_sent.find(player_id);
if (it != tick_sent.end() && it->second == tick_got) {
return false;
}
tick_sent[player_id] = tick_got;
return true;
}
void build_and_send_packet(AetherHuntProtocol::muxer_header& header, predators, preys, cells) {
const size_t data_size = sizeof(AetherHuntProtocol::muxer_header)
+ header.number_of_predators * sizeof(AetherHuntProtocol::predator)
+ header.number_of_prey * sizeof(AetherHuntProtocol::prey)
+ header.number_of_cells * sizeof(protocol::base::net_tree_cell);
void* data = malloc(data_size);
if (data == nullptr) {
return false;
}
uint8_t* predators_ptr, prey_ptr, cell_ptr = /* boring ptr math */
std::memcpy(data, &header, sizeof(AetherHuntProtocol::muxer_header));
write_predators(predators, predators_ptr);
write_prey(preys, prey_ptr);
write_cell(cells, cell_ptr);
aether::netcode::connection_push_packet(connection, muxer, 0, data, data_size);
free(data);
}
};