Persistence
Manage persistent worlds with 24/7 always online deployments on Private Fleets.
Many genres (MMOs, Sandboxes, Social Games) leverage persistent worlds to let players:
meet and socialize with new friends; to nurture organic player communities,
explore a living open world filled with user-generated content placed by players,
engage in epic raid battles lasting hours with large groups or entire guilds.
Explore strategies to provide the best possible player experience, keep cost under control, and remove player frustration due to outages or rollbacks.
Alternatively, see Match-Bound cloud orchestration to utilize fractional vCPU pricing.
✔️ Preparation
To enable persistent, uninterrupted 24/7 always online deployments:
Create a new App Version (or update an existing one).
With Dashboard - choose "persistent" instead of specifying max. duration.
With API - specify
"max_duration": -1in your app version request.
Create a private fleet in dashboard and schedule hosts.
Creating the fleet is free of charge, do this first to preview locations and details.
Choose Virtual Machines (Performance) or Bare Metal (Overdrive) specifications.
You will be prompted to confirm the final price before scheduling hosts.
Deploy new servers with Server Browser or custom integrations.
Server Browser starts servers according to configured Scaling Policies.
Custom integrations should start servers using private fleet API.
Test your server scaling and termination process to verify reliability of your cost controls. Server state stored in memory will be lost once the deployment is stopped, see 💭 Configuration & State.
If you need help, please reach out to us over Discord. For live games support see our ticketing system.
🔑 Server Ownership
Pros and cons of modern vs traditional ownership models with edge computing.
Studio Hosting
Servers traditionally managed by the studio, recoup hosting cost from game revenue.
👍 Advantages
No added player fees - cost of hosting is recouped from studio revenue.
Strong client/server compatibility with loose coupling of client/server/service.
More resilient to cheating and abuse due to game's closed source code.
👎 Disadvantages
Restricted community modding support to ensure server integrity and fairness.
Community Servers
Let your players fund their own servers on Edgegap, and unlock hosting revenue which would otherwise flow to third party hosting services lacking user experience insight.
👍 Advantages
Enhance modding support through curated mods and versions.
Improve your game iteratively through direct collaboration with community.
Build community trust with improved game longevity and self-service model.
👎 Disadvantages
Development investment required for modding support and compatibility.
More operation efforts to manage community and payment systems.
Increased risk of reverse engineering due to exposing game internals.
🥛 Capacity & Scaling
Learn advanced techniques to optimize server cost and quality of service.
Starting your Deployments manually, pasting URL and ports will not cut it for a live game.
Automate popular game flows for managing sessions and scaling on demand with either:
Shorter Rounds
On-Demand Matches
Skill Rating and/or Custom Rules
Persistent or Rounds
Social Regional Hubs
Auto-Assign and/or Custom Search
Custom Backend:
Migrate Live Games
Capacity
Deployments don't track or manage active player connections after you 1. Start a Deployment to give you absolute control and freedom to implement any design.
Implement capacity management to ensure your servers:
control hosting cost - benchmark and optimize server resource usage per player,
provide consistent experience - keep CCU per server within safe range.
See Server Browser for automated capacity handling with our managed service.
Tips: if you choose to develop a custom session and capacity orchestrator.
Procedure must prevent competing for capacity with concurrent reservations.
Send frequent heartbeat to your orchestrator to synchronize server state.
Remove stale, crashed, or stopped servers from discovery process quickly.
Time out capacity reservations if players don't connect within alloted time.
Disconnect clients and release capacity when player inactivity detected.
Amount of reserved deployment resources can't be changed during runtime. Scale horizontally with new server instances utilizing adjusted application versions requiring more CPU or memory resources.
Scale Up
Smart scaling strategies prevent player queue times and minimize idle server cost.
We recommend scaling resources around concurrent players instead of machine load (CPU & RAM), since fluctuations in load may result in unpredictable availability.
Scaling doesn't require "guesstimating" regional traffic or server cost. Schedule Private Fleets capacity for low tide and automatically overflow to cloud during unexpected traffic peaks.
Enable Active Caching in your App Version to deploy servers within seconds.
Scale Down
Shutting down servers without caution may impact player experience negatively. Consider these factors and test changes before releasing:
Is your detection of player inactivity / disconnection reliable?
Is absence of player input reliable? Players often use bots and macros to fake activity and avoid getting kicked, resulting in queue wait time upon reconnect.
Are there any other indicators of activity which are harder to fake?
Is there a game design workaround to mitigate impact/motivation to use bots?
Can you restart servers easily and quickly, without major rollbacks?
Your server may require some time to restart and restore state. Does state restoration incurr any additional data transfer or service cost with your game backend?
Can you hide server loading with a minigame/lobby to keep players engaged?
Are players bound to specific server instances or can they migrate easily?
How does connecting to a different server affect player's account, purchase history, social experience, progression, inventory, and general player retention?
Review your Recovery Objectives and ensure critical data isn't lost.
Transparency and community management do wonders if there's a downtime.
💭 Configuration & State
Define server seed parameters, and manage player/server state reliably.
Configuration Management
Configuration or Seed refers to initial data passed to server during deployment:
environment-specific injected variables:
e.g. client/server version compatibility data,
third party integration parameters, keys, and secrets.
Configuration is immutable. Read during server startup, no modifications during runtime.
State Management
State refers to runtime data, result of previous player actions and server events:
player connection, player-controlled state changes (e.g. Pawn),
property changes contained in the level/scene (e.g. Actor, Game Object),
changes related to game mode, game state, or game scene information.
State data changes frequently. Synchronization occurs many times per second (tick rate).
Frequent state backups help prevent data loss in case of unexpected crashes/issues:
asynchronous updates with a 3rd party game backend API, e.g. Nakama by Heroic Labs,
periodic server heartbeat transferring serialized state files to cloud object storage.
Stateful components usually have a clear designated owner, either server or a player.
Server Owned Objects
Server owned objects can be manipulated only by server. Connected players have limited read access to server owned objects.
Server owned objects can be restored by a replacement server (state migration). They are typically not shared between concurrently running servers (req. server meshing).
Player Owned Objects
Player owned objects can be manipulated both by players and server. Assigning ownership of persistent world objects to players can make storage and state migration easier.
Back up player owned objects on player device or game backend between sessions. Encrypt player state and/or upload state checksums to detect unauthorized modifications.
Recovery Objectives
Some data categories may be more sensitive to data loss and recovery time.
We highly recommend discussing the following amongst your team:
Categories of data handled in your game clients, servers, and game backend.
Impact of data loss in each category, to your players and to your business.
Recovery Point Objective - acceptable amount of data loss before serious harm.
Recovery Time Objective - how quickly does the system need to recover.
Review our simplified recovery objective assessment example below:
account, subscription, and purchases
🔥 5 min
🔥 30 min
progression, inventory, and skill rating
🔥 5 min
🔥60 min
moderation, performance, and error tracking
⚠️ 30 mins
⚠️ 8 hr
social features, chat history, behavioral analytics
⏬ 24 hr
⏬ 72 hr
Prepare a disaster recovery playbook ahead of time for your most critical categories.
Server crash handling depends on your process restart policy. Server state may be lost.
👀 Observability
Long running persistent servers bring new observability challenges, specifically detecting anomalies in monitoring, logging, and bug tracking. We strongly recommend implementing alerts for server restarts to gain traceability and additional oversight for uptime.
Connect your Endpoint Storage to save deployment logs, otherwise they will be deleted!
Last updated
Was this helpful?