# Persistence Learn how to manage persistent worlds with 24/7 always online deployments on [private-fleets](https://docs.edgegap.com/learn/orchestration/private-fleets "mention"). Many genres (MMOs, Sandboxes, Social Games) leverage persistent worlds to let players: * meet and socialize with new friends; to nurture organic player communities, * explore a living open world filled with user-generated content placed by players, * engage in epic raid battles lasting hours with large groups or entire guilds. Explore strategies to **provide the best possible player experience, keep cost under control, and remove player frustration due to outages or rollbacks**. Enhance the traditional server model by bringing the advantages of edge computing packaged for easy use by game developers. {% hint style="success" %} Alternatively, see [#match-bound](https://docs.edgegap.com/learn/deployments#match-bound "mention") orchestration to utilize fractional vCPU pricing. {% endhint %} ## ✔️ Preparation To enable persistent, uninterrupted 24/7 always online deployments: * [create a new (or update an existing one) App Version with our API](https://docs.edgegap.com/docs/api/versioning#post-v1-app-app_name-version), * specify `"max_duration": -1` to prevent automated termination after 24h, * Use [private fleet deploy API](https://docs.edgegap.com/docs/api/dedicated-servers#post-private-fleets-deploy) to start [standby servers](https://docs.edgegap.com/learn/deployments#regional-standby) with [persistence](https://docs.edgegap.com/learn/orchestration/persistence "mention"). * Choose between Virtual Machines (Performance) or Bare Metal (Overdrive) specifications. {% hint style="warning" %} **Test your server scaling and** [**termination process**](https://docs.edgegap.com/learn/deployments#id-5.-deployment-stopped) **to verify reliability of your cost controls. Server state stored in memory will be lost once the deployment is stopped, see** [#configuration-and-state](#configuration-and-state "mention"). {% endhint %} {% hint style="info" %} If you need help, [please reach out to us over Discord](https://discord.gg/MmJf8fWjnt). For live games support see our [ticketing system](https://edgegap.atlassian.net/servicedesk/customer/portal/3). {% endhint %} ## 🔑 Server Ownership Explore pros and cons of modern and traditional ownership models with an edge computing twist. ### Studio Hosting Server hosting is traditionally managed by the studio, covering cost of hosting from game revenue. 👍 **Advantages** * transparent product pricing - cost of hosting is covered by license/subscription of player, * strong client/server compatibility with loose coupling of clients, services, and scaling, * more resilient to cheating and reverse engineering due to closed source nature of servers. 👎 **Disadvantages** * community modding support is limited to ensure server integrity and stability. ### Community Servers Let your players host and fund their own servers and remove need for third party rental services. Funnel revenue through your studio instead of third party lacking insight into quality of end user experience. 👍 **Advantages** * enhanced modding support through curated list of modded [application-and-versions](https://docs.edgegap.com/learn/orchestration/application-and-versions "mention"), * improved player feedback loop due to closer collaboration with community, * reduced financial risk due to players covering cost of hosting. 👎 **Disadvantages** * more operations for studio - moderating player requests and collecting payments, * weaker client/server compatibility due to increased number of modded versions, * prone to cheaters due to distributed codebase and possibility of reverse engineering. ## 🥛 Capacity & Scaling Learn advanced techniques to optimize server availability, hosting cost, and quality of service. ### Capacity **Deployments don't track or manage active player connections** after you [#id-1.-start-a-deployment](https://docs.edgegap.com/learn/deployments#id-1.-start-a-deployment "mention") to give you absolute control and freedom to implement any design. Implement capacity management to ensure your servers: * maximize cost savings - [benchmark](https://docs.edgegap.com/learn/deployments#container-metrics) and utilize server resources efficiently, * provide smooth gameplay - prevent overloading servers with too many concurrent players, * prevent bad reviews due to crashes - catch and handle unexpected exceptions. To ensure efficient server capacity management: * release player slots if [players matched to game server](https://docs.edgegap.com/matchmaking/matchmaker-in-depth#player-tickets) don't connect within a few seconds, * frequently send a minimal heartbeat message from clients to server to keep track of activity, * disconnect clients and release player slots if no activity is detected for several seconds, * prevent players from being added to servers with full capacity and no available player slots. {% hint style="success" %} See [Broken link](https://docs.edgegap.com/learn/orchestration/broken-reference "mention") for automated capacity handling with our managed service. {% endhint %} {% hint style="warning" %} **Amount of reserved deployment resources can't be changed during runtime.** Scale horizontally with new server instances utilizing adjusted [application versions](https://docs.edgegap.com/learn/orchestration/application-and-versions) requiring more CPU or memory resources. {% endhint %} ### Scale Up **Scaling persistent servers doesn't require "guesstimating"** regional traffic or server cost. Reserve [private-fleets](https://docs.edgegap.com/learn/orchestration/private-fleets "mention") capacity for [low tide](#user-content-fn-1)[^1] and automatically overflow to cloud during unexpected peaks. {% hint style="success" %} **Enable** [**Active Caching in your App Version**](https://docs.edgegap.com/learn/application-and-versions#active-caching) **to deploy servers within seconds.** {% endhint %} Implement server scaling strategies to: * enable large scale hosting while carefully protecting against abuse, * minimize wasted server cost due to empty standby servers, * prevent long queue times by responding to increased player demand quickly.

Auto-scaling Reference Architecture

#### Integration Key Points 1. Clients integrate Scaling Authority - [matchmaking](https://docs.edgegap.com/learn/matchmaking "mention"), [server-browser](https://docs.edgegap.com/learn/server-browser "mention"), or a custom solution. 1. Discover other players, reserve capacity in running servers, or request new servers if needed. 2. Scaling Authority assigns running servers or starts new servers to serve players. 3. Server notifies Scaling Authority in real time about server start/stop and player connection changes. 1. Scaling Authority deletes (expires) stale records of unresponsive (crashed) servers. 4. Clients connect and establish game sessions with Server directly, proceed to gameplay. **We strongly recommend scaling based on number of connections instead of physical load** (CPU & RAM), since momentary fluctuations in physical load may result in unpredictable availability. {% hint style="warning" %} **Amount of reserved deployment resources can't be changed during runtime.** Scale horizontally with new server instances utilizing adjusted [application versions](https://docs.edgegap.com/learn/orchestration/application-and-versions) requiring more CPU or memory resources. {% endhint %} ### Scale Down Efficient scale-down policies are key to optimizing cost, but [shutting down servers](https://docs.edgegap.com/docs/api/dedicated-servers#v1-self-stop-request_id-access_point_id) without caution may impact player experience negatively. **Consider these factors and test changes before releasing:** **Is your detection of player activity / disconnection reliable?** * Does absence of input reliably indicate player inactivity? Players often use bots, macros, and other techniques to fake activity and maintain active connection to avoid queue times. * Are there any actions taken by active players often which are hard to fake? * Is using bots or macros an issue or a feature with [#studio-hosting](#studio-hosting "mention") servers? **Is shutting off servers easily and quickly reversible (scale back up)?** * Once reaching [#id-3.-deployment-ready](https://docs.edgegap.com/learn/deployments#id-3.-deployment-ready "mention"), your server may require additional time to perform engine initialization and [#state-management](#state-management "mention") (restoration of state). Do you incur any additional costs for compute or data transfer with game services? Does this wait time impact player experience? * Can you hide server loading with a loading scene, mini-game, a lobby, or through other means? **Are players bound to specific server instances or can they migrate easily?** * How does connecting to a different server influence player's account, purchase history, social experience, progression, inventory, and other gameplay aspects? * Review your [#recovery-objectives](#recovery-objectives "mention") and ensure critical data isn't lost. * Implement automated methods or player tools for restoring critical data. * Provide human support and communicate with your community about outages and issues. ## 🔎 Discoverability To find active servers accepting new players, implement one or more discovery methods: * Let players browse servers and pick from a list with [server-browser](https://docs.edgegap.com/learn/server-browser "mention"). * Start new games when sufficient players join [matchmaking](https://docs.edgegap.com/learn/matchmaking "mention"). * Add players to [replace leavers in existing matches with backfill](https://docs.edgegap.com/matchmaking/matchmaker-in-depth#backfill). * [Implement custom or third party game backend to discover servers](https://docs.edgegap.com/advanced-features/managed-clusters#nakama-by-heroic-labs). ## 💭 Configuration & State Integrate services to define initial server requirements and manage player and server state. ### Configuration Management Configuration refers to the **initial data passed to your server during deployment:** * [environment-specific injected variables](https://docs.edgegap.com/learn/application-and-versions#other-parameters-optional): * e.g. client/server version compatibility data, * [server location](https://docs.edgegap.com/learn/deployments#arbitrium_deployment_location-detailed-information-about-deployment-location), [server ports](https://docs.edgegap.com/learn/deployments#arbitrium_ports_mapping-detailed-information-about-your-internal-and-external-ports) and [other server information](https://docs.edgegap.com/learn/deployments#injected-environment-variables), * [matchmaking information](https://docs.edgegap.com/matchmaking/matchmaker-in-depth#injected-environment-variables) or [custom deployment variables](https://docs.edgegap.com/learn/deployments#custom-variables), * third party integration parameters, keys, and secrets. {% hint style="info" %} **Configuration is immutable** - it's read once after starting your server and doesn't change later on. {% endhint %} ### State Management State refers to data describing the **result of a series of previous player actions and server events:** * player connection, player-controlled state changes (e.g. [Pawn](https://dev.epicgames.com/documentation/en-us/unreal-engine/pawn-in-unreal-engine)), * changes related to objects contained in the level/scene (e.g. [Actor](https://dev.epicgames.com/documentation/en-us/unreal-engine/actors-in-unreal-engine), [Game Object](https://docs.unity3d.com/6000.0/Documentation/Manual/GameObjects.html)), * changes related to [game mode](https://dev.epicgames.com/documentation/en-us/unreal-engine/game-mode-and-game-state-in-unreal-engine#gamemodes), [game state](https://dev.epicgames.com/documentation/en-us/unreal-engine/game-mode-and-game-state-in-unreal-engine#gamestate), or [game scene](https://docs.unity3d.com/6000.0/Documentation/Manual/CreatingScenes.html) information. {% hint style="info" %} **State data changes frequently.** Clients are selectively updated on relevant information by server authority. {% endhint %} {% hint style="success" %} **Perform frequent state backups to prevent data loss** in case of unexpected client or server issues: * asynchronously in real time using a third party service, e.g. [Nakama by Heroic Labs](https://docs.edgegap.com/advanced-features/managed-clusters#nakama-by-heroic-labs), * on client/server startup or shutdown, as deserialized state files in [cloud object storage](https://www.linode.com/products/object-storage/). {% endhint %} Game objects typically designate an owner who controls them, this can be either server or a player. #### Server Owned Objects Server owned objects can be manipulated only by server. Connected players have limited read access to server owned objects. Server owned objects are usually not shared with other servers. {% hint style="success" %} Server owned objects can be **loaded by a replacement server in case of unexpected server crash**. Use [deployment ID](https://docs.edgegap.com/learn/deployments#basic-information) to identify your new save file when first launched, and to store server owned object state. {% endhint %} #### Player Owned Objects Player owned objects can be manipulated both by players and server. Assigning ownership of persistent objects to players makes migration to other servers easier later on. {% hint style="success" %} **Back up state of player owned objects on player device or a game backend** between play sessions. {% endhint %} {% hint style="info" %} Prevent cheating by validating changes with server authority. Authority and ownership can be separate. {% endhint %} ### Recovery Objectives In case of issues, some categories of data may be more sensitive to data loss, for example: * account, subscription, purchase, and microtransaction data - **critical**, * progression, achievement, leaderboards, and inventory data - **important**, * cheat detection, moderation, performance, and error tracking data - **important**, * player behavior, social, chat data - **low importance**. {% hint style="success" %} **Implement restoring purchases from independent transaction history source** for best experience. {% endhint %} We highly recommend discussing the following amongst your team: * categories of data handled in your game clients and servers, * importance and sensitivity of each category for your business and players, * Recovery Point Objective (RPO) - acceptable amount of data loss before serious harm occurs, * Recovery Time Objective (RTO) - acceptable amount of downtime before serious harm occurs. {% hint style="danger" %} **Unexpected crash of server will cause your server to restart automatically.** [Server state may be lost](#state-management). {% endhint %} ## 👀 Observability Long running persistent servers bring new observability challenges, specifically detecting anomalies in monitoring, logging, and bug tracking. We **strongly recommend implementing alerts for server restarts** to gain more visibility over issues. Our [endpoint-storage](https://docs.edgegap.com/docs/endpoint-storage "mention") **log integration** **only transfers logs** **after** [#id-5.-deployment-stopped](https://docs.edgegap.com/learn/deployments#id-5.-deployment-stopped "mention"), add custom logs and bug tracking (such as [Sentry](https://sentry.io/welcome/)) to troubleshoot partial failures and bugs. {% hint style="success" %} Consider reserving [private-fleets](https://docs.edgegap.com/learn/orchestration/private-fleets "mention") standby capacity for games with predictable traffic patterns. {% endhint %} [^1]: lowest CCU during the day