> For the complete documentation index, see [llms.txt](https://docs.edgegap.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.edgegap.com/learn/orchestration/persistence.md). # Persistence Manage persistent worlds with 24/7 always online deployments on [Private Fleets](/learn/orchestration/private-fleets.md). Many genres (MMOs, Sandboxes, Social Games) leverage persistent worlds to let players: * meet and socialize with new friends; to nurture organic player communities, * explore a living open world filled with user-generated content placed by players, * engage in epic raid battles lasting hours with large groups or entire guilds. Explore strategies to **provide the best possible player experience, keep cost under control, and remove player frustration due to outages or rollbacks**. {% hint style="success" %} Alternatively, see [Deployments](/learn/orchestration/deployments.md#match-bound) cloud orchestration to utilize fractional vCPU pricing. {% endhint %} ## ✔️ Preparation To enable persistent, uninterrupted 24/7 always online deployments: 1. [Create a new App Version (or update an existing one).](https://app.edgegap.com/application-management/applications/list) * With Dashboard - choose "persistent" instead of specifying max. duration. * With API - specify `"max_duration": -1` in your app version request. 2. [Create a private fleet in dashboard and schedule hosts.](https://app.edgegap.com/private-fleet-management/private-fleets/list) 1. Creating the fleet is free of charge, do this first to preview locations and details. 2. Choose Virtual Machines (Performance) or Bare Metal (Overdrive) specifications. 3. You will be prompted to confirm the final price before scheduling hosts. 3. Deploy new servers with [Server Browser](/learn/server-browser.md) or custom integrations. 1. Server Browser starts servers according to configured [Scaling Policies](/learn/server-browser.md#automated-scaling). 2. Custom integrations should [start servers using private fleet API](https://docs.edgegap.com/learn/orchestration/pages/58CVGf4IKRrTIMgc5xr5#id-3.-deploy-servers). {% hint style="warning" %} **Test your server scaling and** [**termination process**](https://docs.edgegap.com/learn/orchestration/pages/0UXQAhtFuL0FkdoUmYTh#id-5.-deployment-stopped) **to verify reliability of your cost controls. Server state stored in memory will be lost once the deployment is stopped, see** [#configuration-and-state](#configuration-and-state "mention"). {% endhint %} {% hint style="info" %} If you need help, [please reach out to us over Discord](https://discord.gg/MmJf8fWjnt). For live games support see our [ticketing system](https://edgegap.atlassian.net/servicedesk/customer/portal/3). {% endhint %} ## 🔑 Server Ownership Pros and cons of modern vs traditional ownership models with edge computing. ### Studio Hosting Servers traditionally managed by the studio, recoup hosting cost from game revenue. 👍 **Advantages** * No added player fees - cost of hosting is recouped from studio revenue. * Strong client/server compatibility with loose coupling of client/server/service. * More resilient to cheating and abuse due to game's closed source code. 👎 **Disadvantages** * Restricted community modding support to ensure server integrity and fairness. ### Community Servers Let your players fund their own servers on Edgegap, and unlock hosting revenue which would otherwise flow to third party hosting services lacking user experience insight. 👍 **Advantages** * Enhance modding support through curated mods and versions. * Improve your game iteratively through direct collaboration with community. * Build community trust with improved game longevity and self-service model. 👎 **Disadvantages** * Development investment required for modding support and compatibility. * More operation efforts to manage community and payment systems. * Increased risk of reverse engineering due to exposing game internals. ## 🥛 Capacity & Scaling Learn advanced techniques to optimize server cost and quality of service.

Auto-scaling Reference Architecture

{% hint style="warning" %} **Starting your Deployments manually, pasting URL and ports will not cut it for a live game.** {% endhint %} Automate popular game flows for managing sessions and scaling on demand with either: {% columns %} {% column width="33.33333333333333%" %} [Matchmaking](/learn/matchmaking.md): * Shorter Rounds * On-Demand Matches * Skill Rating and/or\ Custom Rules {% endcolumn %} {% column width="33.33333333333333%" %} [Server Browser](/learn/server-browser.md): * Persistent or Rounds * Social Regional Hubs * Auto-Assign and/or\ Custom Search {% endcolumn %} {% column width="33.33333333333333%" %} Custom Backend: * Migrate Live Games * [Deploy with v2 API](/docs/api/dedicated-servers.md) * [Observe Webhooks](/learn/orchestration/deployments.md#webhooks) {% endcolumn %} {% endcolumns %} ### Capacity **Deployments don't track or manage active player connections** after you [/pages/0UXQAhtFuL0FkdoUmYTh#id-1.-start-a-deployment](https://docs.edgegap.com/learn/orchestration/pages/0UXQAhtFuL0FkdoUmYTh#id-1.-start-a-deployment "mention") to give you absolute control and freedom to implement any design. Implement capacity management to ensure your servers: * **control hosting cost** - [benchmark](/learn/orchestration/deployments.md#container-metrics) and optimize server resource usage per player, * **provide consistent experience** - keep CCU per server within safe range. {% hint style="success" %} See [Server Browser](/learn/server-browser.md) for automated capacity handling with our managed service. {% endhint %}
Tips: if you choose to develop a custom session and capacity orchestrator. * Procedure must prevent competing for capacity with concurrent reservations. * Send frequent heartbeat to your orchestrator to synchronize server state. * Remove stale, crashed, or stopped servers from discovery process quickly. * Time out capacity reservations if players don't connect within alloted time. * Disconnect clients and release capacity when player inactivity detected.
{% hint style="warning" %} **Amount of reserved deployment resources can't be changed during runtime.** Scale horizontally with new server instances utilizing adjusted [application versions](/learn/orchestration/application-and-versions.md) requiring more CPU or memory resources. {% endhint %} ### Scale Up Smart scaling strategies **prevent player queue times and minimize idle server cost**. **We recommend scaling resources around concurrent players instead of machine load** (CPU & RAM), since fluctuations in load may result in unpredictable availability. {% hint style="info" %} **Scaling doesn't require "guesstimating" regional traffic or server cost**. Schedule [Private Fleets](/learn/orchestration/private-fleets.md) capacity for [low tide](#user-content-fn-1)[^1] and automatically overflow to cloud during unexpected traffic peaks. {% endhint %} {% hint style="success" %} Enable [Active Caching in your App Version](/learn/orchestration/application-and-versions.md#active-caching) to deploy servers within seconds. {% endhint %} ### Scale Down Shutting down servers without caution may impact player experience negatively. **Consider these factors and test changes before releasing:** **Is your detection of player inactivity / disconnection reliable?** * Is absence of player input reliable? Players often use bots and macros to fake activity and avoid getting kicked, resulting in queue wait time upon reconnect. * Are there any other indicators of activity which are harder to fake? * Is there a game design workaround to mitigate impact/motivation to use bots? **Can you restart servers easily and quickly, without major rollbacks?** * Your server may require some time to restart and restore state. Does state restoration incurr any additional data transfer or service cost with your game backend? * Can you hide server loading with a minigame/lobby to keep players engaged? **Are players bound to specific server instances or can they migrate easily?** * How does connecting to a different server affect player's account, purchase history, social experience, progression, inventory, and general player retention? * Review your [#recovery-objectives](#recovery-objectives "mention") and ensure critical data isn't lost. * Transparency and community management do wonders if there's a downtime. ## 💭 Configuration & State Define server seed parameters, and manage player/server state reliably. ### Configuration Management Configuration or Seed refers to **initial data passed to server during deployment:** * [environment-specific injected variables](/learn/orchestration/application-and-versions.md#other-parameters-optional): * e.g. client/server version compatibility data, * [server location](/learn/orchestration/deployments.md#arbitrium_deployment_location-detailed-information-about-deployment-location), [server ports](/learn/orchestration/deployments.md#arbitrium_ports_mapping-detailed-information-about-your-internal-and-external-ports) and [other server information](/learn/orchestration/deployments.md#injected-environment-variables), * [matchmaking information](/learn/matchmaking/matchmaker-in-depth.md#injected-environment-variables) or [custom deployment variables](/learn/orchestration/deployments.md#custom-variables), * third party integration parameters, keys, and secrets. {% hint style="info" %} **Configuration is immutable.** Read during server startup, no modifications during runtime. {% endhint %} ### State Management State refers to runtime data, **result of previous player actions and server events:** * player connection, player-controlled state changes (e.g. [Pawn](https://dev.epicgames.com/documentation/en-us/unreal-engine/pawn-in-unreal-engine)), * property changes contained in the level/scene (e.g. [Actor](https://dev.epicgames.com/documentation/en-us/unreal-engine/actors-in-unreal-engine), [Game Object](https://docs.unity3d.com/6000.0/Documentation/Manual/GameObjects.html)), * changes related to [game mode](https://dev.epicgames.com/documentation/en-us/unreal-engine/game-mode-and-game-state-in-unreal-engine#gamemodes), [game state](https://dev.epicgames.com/documentation/en-us/unreal-engine/game-mode-and-game-state-in-unreal-engine#gamestate), or [game scene](https://docs.unity3d.com/6000.0/Documentation/Manual/CreatingScenes.html) information. {% hint style="info" %} **State data changes frequently.** Synchronization occurs many times per second (tick rate). {% endhint %} {% hint style="success" %} **Frequent state backups help prevent data loss** in case of unexpected crashes/issues: * asynchronous updates with a 3rd party game backend API, e.g. [Nakama by Heroic Labs](/learn/advanced-features/managed-clusters.md#nakama-by-heroic-labs), * periodic server heartbeat transferring serialized state files to [cloud object storage](https://www.linode.com/products/object-storage/). {% endhint %} Stateful components usually have a clear designated owner, either server or a player. #### Server Owned Objects Server owned objects can be manipulated only by server. Connected players have limited read access to server owned objects. {% hint style="success" %} Server owned objects can be **restored by a replacement server (state migration)**. They are typically not shared between concurrently running servers (req. server meshing). {% endhint %} #### Player Owned Objects Player owned objects can be manipulated both by players and server. Assigning ownership of persistent world objects to players can make storage and state migration easier. {% hint style="success" %} **Back up player owned objects on player device or game backend** between sessions. Encrypt player state and/or upload state checksums to detect unauthorized modifications. {% endhint %} ### Recovery Objectives Some data categories may be more sensitive to data loss and recovery time. We highly recommend discussing the following amongst your team: * Categories of data handled in your game clients, servers, and game backend. * Impact of data loss in each category, to your players and to your business. * Recovery Point Objective - acceptable amount of data loss before serious harm. * Recovery Time Objective - how quickly does the system need to recover. Review our simplified recovery objective assessment example below:
Data CategoryRPORTO
account, subscription, and purchases🔥 5 min🔥 30 min
progression, inventory, and skill rating🔥 5 min🔥60 min
moderation, performance, and error tracking⚠️ 30 mins⚠️ 8 hr
social features, chat history, behavioral analytics⏬ 24 hr⏬ 72 hr
{% hint style="success" %} Prepare a disaster recovery playbook ahead of time for your most critical categories. {% endhint %} {% hint style="danger" %} **Server crash handling depends on your** [**process restart policy**](/learn/orchestration/application-and-versions.md#safety-guardrails)**.** [Server state may be lost](/learn/orchestration/persistence.md#state-management). {% endhint %} ## 👀 Observability Long running persistent servers bring new observability challenges, specifically detecting anomalies in monitoring, logging, and bug tracking. We **strongly recommend implementing alerts for server restarts** to gain traceability and additional oversight for uptime. {% hint style="warning" %} Connect your [Endpoint Storage](/docs/endpoint-storage.md) to save deployment logs, otherwise they will be deleted! {% endhint %} {% hint style="info" %} Add custom logs and bug tracking ([Sentry](https://sentry.io/welcome/), [Bugsnag](https://www.bugsnag.com/)) to troubleshoot partial failures. {% endhint %} [^1]: lowest CCU during the day