# Managed Clusters

Learn about Managed Clusters and get started quickly with custom backend solutions.

## ✔️ Introduction

Managed Clusters make hosting self-managed game services and game backend easy and fast. You prepare the service image and we provide a high-availability, resilient cloud environment to run them:

* player authentication,
* data storage - accounts, progression, inventory, rewards, ...
* social services - chat, clans, leaderboards, tournaments, ...
* custom matchmaking - using [#advanced-matchmaker](#advanced-matchmaker "mention"), [#nakama-by-heroic-labs](#nakama-by-heroic-labs "mention"), ...
* serverless compute - managed [functions as a service](https://github.com/openfaas/faas) (alt. cloudscript, lambda), ...

{% hint style="success" %}
See [matchmaking](https://docs.edgegap.com/learn/matchmaking "mention") and [server-browser](https://docs.edgegap.com/learn/server-browser "mention") to get started with game client integration.
{% endhint %}

Private clusters ensure your services have **dedicated compute to serve your players 24/7**.

We currently offer [3 private cluster tiers](https://edgegap.com/resources/pricing#managed-infrastructure) to cater to everybody’s needs:

<table><thead><tr><th width="160">Tier</th><th align="right">Hobbyist Tier</th><th align="right">Studio Tier</th><th align="right">Enterprise Tier</th></tr></thead><tbody><tr><td>Best Suited For</td><td align="right">enthusiasts,<br>solo developers</td><td align="right">commercial releases</td><td align="right">high-traffic launches</td></tr><tr><td>Resources</td><td align="right">1 vCPU + 2GB RAM</td><td align="right">6 vCPU + 12GB RAM</td><td align="right">18 vCPU + 48GB RAM</td></tr><tr><td>Redundancy</td><td align="right">1x virtual node</td><td align="right">3x virtual nodes</td><td align="right">3x virtual nodes</td></tr><tr><td>Rate Limit (req/s)</td><td align="right">200</td><td align="right">750</td><td align="right">2,000</td></tr><tr><td>Price, hourly</td><td align="right">$0.0312</td><td align="right"> $0.146</td><td align="right">$0.548</td></tr><tr><td><strong>Price, 30 days</strong><br>(nonstop usage)</td><td align="right"><strong>$22.464</strong></td><td align="right"><strong>$105.12</strong></td><td align="right"><strong>$394.56</strong></td></tr></tbody></table>

{% hint style="info" %}
Our cluster machines use AMD/Intel CPUs with clock speed 2.4 - 3.2 GHz. Reach out on [Community Discord](https://discord.gg/MmJf8fWjnt) to coordinate load tests and to ensure your server has sufficient resources available.
{% endhint %}

## 🛠️ Developer Tools

If you see an opportunity for improvement, please let us know in our [Community Discord](https://discord.gg/NgCnkHbsGp).

We hope you will enjoy a smooth experience. 🚀

### Docker

To help make your server reliable, we use [Docker](https://www.docker.com/) - virtualization software to ensuring that all of your server code dependencies down to the operating system level are going to be always exactly the same, no matter how or where the server is launched.

{% hint style="info" %}
We recommend watching ["Never install locally" (video)](https://www.youtube.com/watch?v=J0NuOlA2xDc\&ab_channel=Coderized). **You DON'T need to use Dockerhub with Docker**.  Docker ≠ Dockerhub. Think of Docker as a programming engine and Dockerhub as it’s App Store.
{% endhint %}

### Kubernetes (K8s)

[Kubernetes](https://kubernetes.io/docs/concepts/overview/), also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications (Docker Images). It groups containers that make up an application into logical units for easy management and discovery.

Edgegap Managed Clusters provide a Kubernetes API for administration purposes.

### K8s Lens

With over 1 million users, [K8s Lens](https://k8slens.dev/) is the most popular Kubernetes IDE in the world. Connect to clusters, explore, gain insights, learn and take an action when needed. Lens provides all the information from your workloads and resources in real-time, always in the right context.

Edgegap Cluster Kubernetes API can be used through Lens or other Kubernetes IDEs.

### Helm Package Manager

[Helm](https://helm.sh/) is the best way to find, share, and use software built for Kubernetes. Helm helps you manage Kubernetes applications - Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Charts are easy to create, version, share, and publish - so start using Helm and stop the copy-and-paste.

[Installing Helm CLI](https://helm.sh/docs/intro/install/) provides developers with a simple interface to manage their cluster packages.

## 🚀 Getting Started

☑️ [Registered for your free Edgegap account](https://app.edgegap.com/auth/register) and upgrade to pay as you go tier to unlock Clusters.

☑️ Navigate to [Managed Clusters](https://app.edgegap.com/cluster-management/clusters/list) page.

☑️ Click on **Create Cluster** first, then input:

* **Label** for your cluster to find it later easily,
* **Cluster Size -** see [#introduction](#introduction "mention").

{% hint style="danger" %}
**We strongly recommend creating separate clusters for your dev & production environments.**
{% endhint %}

☑️ Review estimated cost and click **Create Cluster** to start your new cluster.

☑️ Once the cluster is ready, **click Kubeconfig to download your configuration and credentials** for connecting and administrating your new cluster.

☑️ [Move your kubeconfig file](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) for `kubectl` to find it.

☑️ Lens users: [import your kubeconfig file](https://docs.k8slens.dev/getting-started/add-cluster/#specify-kubeconfig-files).

☑️ Test your cluster connection with command `kubectl get nodes` :

```bash
kubectl get nodes
NAME                            STATUS   ROLES    AGE    VERSION
lke334087-533013-294dcfe70000   Ready    <none>   10m   v1.31.0
lke334087-533013-4e69edc10000   Ready    <none>   10m   v1.31.0
lke334087-533013-50bf39880000   Ready    <none>   10m   v1.31.0
```

🙌 Congratulations, you’ve completed Managed Cluster setup! You may now install your services.

## 📦 Nakama by Heroic Labs

{% hint style="success" %}
See [#getting-started](#getting-started "mention") before setting up your services on Managed Clusters.
{% endhint %}

{% hint style="info" %}
Integrate Edgegap with [Nakama plugin](https://github.com/edgegap/nakama-edgegap) + [Unity plugin](https://github.com/edgegap/edgegap-server-nakama-plugin-unity)! [Reach out for other platforms/features.](https://discord.gg/NgCnkHbsGp)
{% endhint %}

Follow these steps to host your own [Nakama Game Backend](https://heroiclabs.com/docs/nakama/getting-started/) on Managed Clusters:

☑️ Install a **Certificate Manager** to support usage of HTTPS requests from game clients:

```bash
helm repo add jetstack https://charts.jetstack.io --force-update;
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.16.1 --set crds.enabled=true;
```

☑️ Lens: verify installation in Services / Network section, `cert-manager` should be running.

☑️ Write a **Cluster Issuer** file, remember to replace `<YOUR_EMAIL>` below:

<pre class="language-yaml"><code class="lang-yaml">apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <a data-footnote-ref href="#user-content-fn-1">&#x3C;YOUR_EMAIL></a>
    privateKeySecretRef:
      name: <a data-footnote-ref href="#user-content-fn-2">letsencrypt-staging</a>
    solvers:
    - http01:
        ingress:
          class: nginx
          
</code></pre>

{% hint style="danger" %}
Use `letsencrypt-prod` for your production cluster `privateKeySecretRef`.
{% endhint %}

☑️ Create a **Cluster Issuer** in command line:

```bash
kubectl create -f <FILE_PATH>/issuer.yaml
```

☑️ Lens: verify installation in Custom Resources section / `cert-manager.io` - cluster issuer created.

☑️ Install the **nginx Ingress** for receiving client requests and passing them to services in cluster:

```
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace
```

☑️ Lens: verify installation in Services / Network section, `nginx` should be running.

☑️ Create a **DNS record type A** in your DNS provider (e.g. [Cloudflare](https://developers.cloudflare.com/dns/get-started/)), note the URL for later. Your **external IP for the DNS record** can be found in Lens under Services / `ingress-nginx-controller` .

☑️ Verify your DNS is set up correctly by performing a lookup [using DNSchecker](https://dnschecker.org/ns-lookup.php).

☑️ Create file named `values.yaml` with contents (use your own values):

<pre class="language-yaml"><code class="lang-yaml">isProductionEnvironment: true

# The external hostname for the Nakama server - DNS record from last step
externalHostName: <a data-footnote-ref href="#user-content-fn-3">&#x3C;DNS_A_RECORD_URL></a>

nakama:
  # The version of Nakama to deploy.
  # See https://hub.docker.com/r/heroiclabs/nakama/tags for available versions.
  version: 3.26.0
  # Username and password for the Nakama console
  username: <a data-footnote-ref href="#user-content-fn-4">&#x3C;USERNAME></a>
  password: <a data-footnote-ref href="#user-content-fn-5">&#x3C;PASSWORD></a>
</code></pre>

{% hint style="danger" %}
**Replace \<VALUES> above with your own values** in the file above.
{% endhint %}

☑️ Deploy Nakama helm chart:

```bash
helm upgrade --install \
  --namespace nakama --create-namespace -f <FILE_PATH>/values.yaml \
  --version 1.0.0 <RELEASE_NAME> oci://registry-1.docker.io/edgegap/heroiclabs-nakama
```

☑️ Lens: verify installation in Workloads / Deployments section, `nakama` should be running.

✅ **Connect to your Nakama Console** with URL and credentials from `values.yaml` file.

🙌 Congratulations, you’ve completed self-hosted Nakama Game Backend setup!

### Service Updates

Follow these steps to update your service hosted in the Managed Cluster:

☑️ Update your `value.yaml` file with new files.

☑️ Update your helm chart using this command:

```bash
helm upgrade --reuse-values \
  --namespace nakama -f <FILE_PATH>/values.yaml \
  --version 1.0.0 <RELEASE_NAME> oci://registry-1.docker.io/edgegap/heroiclabs-nakama
```

☑️ Reload your changes by closing the updated pods, causing the new helm chart to be used after we automatically restart the pods.

{% hint style="warning" %}
This **update will cause a short downtime** while the pods restart.
{% endhint %}

🙌 Congratulations, you’ve completed Nakama Cluster update!

## 👷 Advanced Matchmaker

{% hint style="success" %}
See [#getting-started](#getting-started "mention") before setting up your services on Managed Clusters.
{% endhint %}

Follow these steps to host your [OpenMatch](https://open-match.dev/site/) on a Managed Cluster.

☑️ Install a **Certificate Manager** to support usage of HTTPS requests from game clients:

```bash
helm repo add jetstack https://charts.jetstack.io --force-update;
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.16.1 --set crds.enabled=true;
```

☑️ Lens: verify installation in Services / Network section, `cert-manager` should be running.

☑️ Write a **Cluster Issuer** file, remember to replace `<YOUR_EMAIL>` below:

<pre class="language-yaml"><code class="lang-yaml">apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <a data-footnote-ref href="#user-content-fn-1">&#x3C;YOUR_EMAIL></a>
    privateKeySecretRef:
      name: <a data-footnote-ref href="#user-content-fn-2">letsencrypt-staging</a>
    solvers:
    - http01:
        ingress:
          class: nginx
          
</code></pre>

{% hint style="danger" %}
Use `letsencrypt-prod` for your production cluster `privateKeySecretRef`.
{% endhint %}

☑️ Create a **Cluster Issuer** in command line:

```bash
kubectl create -f <FILE_PATH>/issuer.yaml
```

☑️ Lens: verify installation in Custom Resources section / `cert-manager.io` - cluster issuer created.

☑️ Install the **nginx Ingress** for receiving client requests and passing them to services in cluster:

```
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace
```

☑️ Lens: verify installation in Services / Network section, `nginx` should be running.

☑️ Create a DNS record type A in your DNS provider (e.g. [Cloudflare](https://developers.cloudflare.com/dns/get-started/)), note the URL for later. Your **external IP for the DNS record** can be found in Lens under Services / `ingress-nginx-controller` .

☑️ Verify your DNS is set up correctly by performing a lookup [using DNSchecker](https://dnschecker.org/ns-lookup.php).

☑️ Create file named `values.yaml` with contents (use your own values):

```yaml
isProductionEnvironment: false

director:
  credential:
    registry: <YOUR_DIRECTOR_REGISTRY>
    username: <YOUR_DIRECTOR_REGISTRY_USERNAME>
    password: <YOUR_DIRECTOR_REGISTRY_PASSWORD>
  image: <DIRECTOR_IMAGE>
  env: {
    "KEY": "VALUE"
  }

mmf:
  credential:
    registry: <MATCHMAKER_FUNCTION_REGISTRY>
    username: <MATCHMAKER_FUNCTION_REGISTRY_USERNAME>
    password: <MATCHMAKER_FUNCTION_REGISTRY_PASSWORD>
  image: <MATCHMAKER_FUNCTION_IMAGE>
  env: {
    "KEY": "VALUE"
  }

frontend:
  credential:
    registry: <FRONTEND_REGISTRY>
    username: <FRONTEND_REGISTRY_USERNAME>
    password: <FRONTEND_REGISTRY_PASSWORD>
  externalHostName: <YOUR_CLOUDFLARE_HOST_NAME> # e.g. exemple.test.com
  image: <FRONTEND_IMAGE>
  env: {
    "KEY": "VALUE"
  }

# Global configurations that are visible to all subcharts
global:
  kubernetes:
    resources:
      requests:
        memory: 100Mi
        cpu: 100m
      limits:
        memory: 100Mi
        cpu: 100m
```

{% hint style="danger" %}
**Replace \<VALUES> above with your own values** in the file above.
{% endhint %}

☑️ Add **Edgegap repository** to your list of repositories:

```bash
helm repo add edgegap-public https://registry.edgegap.com/chartrepo/edgegap-public
```

☑️ Deploy advanced matchmaker helm chart:

```bash
helm upgrade --install \
  --namespace matchmaker --create-namespace -f <FILE_PATH>/values.yaml \
  --version 1.0.1 <RELEASE_NAME> edgegap-public/open-match-edgegap
```

🙌 Congratulations, you’ve completed Advanced Matchmaker setup!

### Service Updates

Follow these steps to update your service hosted in the Managed Cluster:

☑️ Update your `value.yaml` file with new files.

☑️ Update your helm chart using this command:

```bash
helm upgrade --reuse-values \
  --namespace matchmaker -f <FILE_PATH>/values.yaml \
  --version 1.0.1 <RELEASE_NAME> edgegap-public/open-match-edgegap
```

☑️ Reload your changes by closing the updated pods (director, mmf, frontend), causing the new helm chart to be used after we automatically restart the pods.

{% hint style="warning" %}
This **update will cause a short downtime** while the pods restart.
{% endhint %}

🙌 Congratulations, you’ve completed Advanced Matchmaker update!

### Continuous Deployment

Automate updating your services by adding this shell script to your deployment pipeline:

```bash
#!/bin/bash

RELEASE_NAME="<RELEASE_NAME>"
NAMESPACE="matchmaker"  # Change this if you changed the namespace.

helm upgrade --reuse-values -f <FILE_PATH>/value.yaml --namespace $NAMESPACE --version 1.0.1 $RELEASE_NAME edgegap-public/open-match-edgegap

echo "Installing redis-tools"
apt-get update
apt-get install -y redis-tools

DIRECTOR_DEPLOYMENT_NAME="$RELEASE_NAME-director"
MMF_DEPLOYMENT_NAME="$RELEASE_NAME-mmf"
CUSTOM_FRONTEND_DEPLOYMENT_NAME="$RELEASE_NAME-custom-frontend"
REDIS_HOST="$RELEASE_NAME-redis-master"

declare -A replicas

# For each deployment (director, mmf, custom-frontend) stop the pods
for deployment in $DIRECTOR_DEPLOYMENT_NAME $MMF_DEPLOYMENT_NAME $CUSTOM_FRONTEND_DEPLOYMENT_NAME
do
  echo "Stopping pods for deployment: $deployment"
  replicas[$deployment]=$(kubectl get deployment $deployment -o=jsonpath='{.spec.replicas}' --namespace $NAMESPACE)
  kubectl scale deployment/$deployment --replicas=0 --namespace $NAMESPACE
done

# Wait until the pods are terminated
for deployment in $DIRECTOR_DEPLOYMENT_NAME $MMF_DEPLOYMENT_NAME $CUSTOM_FRONTEND_DEPLOYMENT_NAME
do

  echo "Waiting for pods to be terminated for deployment: $deployment"
  kubectl wait --for=delete pod -l app=$deployment --timeout=60s --namespace $NAMESPACE

  # Check if the wait command was successful. If not, exit the script
  if [ $? -ne 0 ]; then
    echo "Failed to wait for pods to be terminated for deployment: $deployment"
    exit 1
  fi
done

# Clean up redis database
echo "Cleaning up redis database"
redis-cli -h $REDIS_HOST flushall

# For each deployment (director, mmf, custom-frontend) rescale the pods to their original count
for deployment in $DIRECTOR_DEPLOYMENT_NAME $MMF_DEPLOYMENT_NAME $CUSTOM_FRONTEND_DEPLOYMENT_NAME
do
  echo "Rescaling pods to ${replicas[$deployment]} for deployment: $deployment"
  kubectl scale deployment/$deployment --replicas=${replicas[$deployment]} --namespace $NAMESPACE
done
```

{% hint style="warning" %}
This **update will cause a short downtime** while the pods restart.
{% endhint %}

### Letsencrypt Certificate Validation (C#)

For some clients, the recommended Letsencrypt certificate validation may fail with error:

```bash
Curl error 60: Cert verify failed. Certificate has expired. UnityTls error code: 7
```

{% hint style="success" %}
Updating your operating system may resolve issues with outdated root certificate authority.
{% endhint %}

As a last resort, game clients may implement a custom certificate handler function:

````csharp
```csharp
public class CustomCertificateHandler : CertificateHandler
{
  private readonly string EXPECTED_CERT = "-----BEGIN CERTIFICATE-----<key>-----END CERTIFICATE-----\r\n";
  protected override bool ValidateCertificate(byte[] certificateData)
  {
    X509Certificate2 certificate = new X509Certificate2(certificateData);
    X509Certificate2 expectedCert = new X509Certificate2(Encoding.ASCII.GetBytes(EXPECTED_CERT));

    using (SHA256 sha256 = SHA256.Create())
    {
      Debug.Log("certificate.Thumbprint: " + certificate.Thumbprint);
      Debug.Log("expectedCert.Thumbprint: " + expectedCert.Thumbprint);

      return certificate.Thumbprint == expectedCert.Thumbprint;
    }
  }
}
```
````

Usage:

```csharp
UnityWebRequest request = UnityWebRequest.Get(...);
request.certificateHandler = new BypassCertificateHandler();
request.SendWebRequest();
request.certificateHandler.Dispose();
```

We recommend storing the `EXPECTED_CERT` value in your own file storage, and retrieving it on runtime, so you can update it without releasing a game client update.

## 🟢 Operations and Observability

### Cluster Tier Changes

Prepare for success and optimize after launch, so you don’t block your players on release day.

{% hint style="warning" %}
&#x20;Changing cluster size requires stopping your cluster. See [blue/green deployment](https://circleci.com/blog/canary-vs-blue-green-downtime/) for zero downtime updates.
{% endhint %}

### Support and Future Updates

**Your success is our priority.** If you'd like to send custom requests, ask for missing critical features , or express any thoughts, [please reach out in our Community Discord](https://discord.gg/MmJf8fWjnt).

[^1]: replace with your email

[^2]: use letsencrypt-prod for production

[^3]: DNS record from last step

[^4]: choose your own admin username

[^5]: choose a secure password for admin
