Helm charts for setting up HPB clusters¶

A Helm repository is provided for setting up a HPB cluster in Kubernetes environments.

Two charts are provided:

• controller: Manages the global controller that creates / updates / deletes the HPB clusters.
• cluster: Defines the settings of a HPB cluster consisting of one or multiple signaling servers and/or signaling proxies.

Requirements¶

• A unique customer id to access the Helm repository.
• Access to a running Kubernetes cluster.
• A CNI plugin to manage additional networks for pods, e.g. Multus CNI. This is necessary for the pods running the Janus WebRTC gateway to expose large port ranges on the public network (which is not possible through Kubernetes services).

Add repository¶

The Helm repository can be added using helm repo add:

$ helm repo add hpb \
      https://packages.struktur.de/nextcloud/spreed/<the-customer-id>/helm/

Afterwards you can check the available charts:

$ helm search repo hpb
NAME          	CHART VERSION	APP VERSION	DESCRIPTION
hpb/cluster   	0.1.0        	0.1.0      	A Helm chart for setting up a Talk HPB cluster.
hpb/controller	0.1.0        	0.1.0      	A Helm chart for installing the Nextcloud Talk ...

Chart controller¶

The controller is installed once in the cluster and manages the HPB clusters created by the cluster chart (see below).

Example (creates the controller as hpb-controller in namespace talk-hpb-controller):

$ helm install hpb-controller hpb/controller \
    --namespace talk-hpb-controller --create-namespace

Usually, no configuration is required, but you might want to select a dedicated node for, change the affinity for the created pod or limit its resources. You can optionally define annotations or labels that should be added.

In case of problems (i.e. PODs not starting correctly or getting re-created), you can enable debug mode in the controller to get a more detailed log that will help tracking down the issues.

You can check available options by running

$ helm show values hpb/controller

Check the Helm documentation on how to customize the chart before installing at https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing

Chart cluster¶

To create a signaling server and/or signaling proxy for a HPB cluster, the cluster chart can be used:

Example (create cluster as cluster1 in the default namespace):

$ helm install cluster1 hpb/cluster --values cluster1.yaml

Clusters need configuration to specify what should be created by the controller and how the services should be setup.

Some basic example configurations are shown below.

You can check all available options by running

$ helm show values hpb/cluster

Most options map to the configuration of the signaling server / proxy, details are available at https://github.com/strukturag/nextcloud-spreed-signaling/blob/master/server.conf.in and https://github.com/strukturag/nextcloud-spreed-signaling/blob/master/proxy.conf.in

Check the Helm documentation on how to customize the chart before installing at https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing

It is usually recommended to use one signaling server per Nextcloud instance and share a cluster of signaling proxies between the servers.

However depending on customer requirements, it might be necessary to have dedicated proxies for each Nextcloud instance (to enforce isolation), or to have multiple signaling servers handling traffic for multiple Nextcloud instances.

More details on the different modes of running a HPB cluster are also available in the Nextcloud documentation at https://portal.nextcloud.com/categories/Partner-Products/Talk-High-Performance-Backend

Simple cluster with signaling server and proxy¶

This will create a signaling server using one Nextcloud instance as backend and a signaling proxy to handle WebRTC traffic.

networks:
  public-net: |
    {
      "cniVersion": "0.3.1",
      "type": "ipvlan",
      "capabilities": { "ips": true },
      "master": "br0",
      "mode": "l2",
      "ipam": {
        "type": "static"
      }
    }

signaling:
  hashKey: "1234567890123456789012345678901234567890123456789012345678901234"
  blockKey: "12345678901234567890123456789012"
  internalSharedSecret: "verysecret"

  backends:
    - url: "https://cloud.domain.tld"
      sharedSecret: "the-shared-secret"

  ingress:
    host: "signaling.domain.tld"
    tlsSecretName: "signaling-domain-tld-tls"
    annotations:
      cert-manager.io/issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx

proxy:
  janus:
    gateways:
      - network:
          name: "public-net"
          publicIP: "10.2.0.100"
          gatewayIP: "10.2.0.1"
        portRange:
          min: 20000
          max: 40000

Signaling server-only cluster using external proxies¶

This will create a signaling server using one Nextcloud instance as backend and an external cluster of signaling proxies.

signaling:
  hashKey: "1234567890123456789012345678901234567890123456789012345678901234"
  blockKey: "12345678901234567890123456789012"
  internalSharedSecret: "verysecret"

  backends:
    - url: "https://cloud.domain.tld"
      sharedSecret: "the-shared-secret"

  ingress:
    host: "signaling.domain.tld"
    tlsSecretName: "signaling-domain-tld-tls"
    annotations:
      cert-manager.io/issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx

  proxy:
    tokenId: "signaling1"
    tokenKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAuV0epw6IvrGpVJxYNqnvlcrLpxSHMwoaXvVPCNvD0jKfi+Zp
      ...snip...
      YW4QGLxpT+kRAPyPSZDz2U+k35cxTDvHUz0Xa75qfFFXL3xvEOqf
      -----END RSA PRIVATE KEY-----
    urls:
      - https://proxy1.hpb.domain.tld
      - https://proxy2.hpb.domain.tld
      - https://proxy3.hpb.domain.tld

Signaling server-only cluster using external proxies configured with etcd¶

This will create a signaling server using one Nextcloud instance as backend and an external cluster of signaling proxies. The available signaling proxies are configured through an etcd cluster.

In the etcd cluster, the prefix /signaling/proxy/server will be monitored for added / removed entries and connections to the signaling proxies will be created or removed accordingly.

To add a new entry, you can execute the following command:

etcdctl put /signaling/proxy/server/server1 '{"address":"https://proxy-01.domain.tld/"}'

This will create a new connection to the signaling proxy at https://proxy-01.domain.tld/.

Removing an entry can be done using this command:

etcdctl del /signaling/proxy/server/server1

This will wait until all existing streams using the proxy are finished and will then remove the connection.

signaling:
  hashKey: "1234567890123456789012345678901234567890123456789012345678901234"
  blockKey: "12345678901234567890123456789012"
  internalSharedSecret: "verysecret"

  backends:
    - url: "https://cloud.domain.tld"
      sharedSecret: "the-shared-secret"

  ingress:
    host: "signaling.domain.tld"
    tlsSecretName: "signaling-domain-tld-tls"
    annotations:
      cert-manager.io/issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx

  proxy:
    tokenId: "signaling1"
    tokenKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAuV0epw6IvrGpVJxYNqnvlcrLpxSHMwoaXvVPCNvD0jKfi+Zp
      ...snip...
      YW4QGLxpT+kRAPyPSZDz2U+k35cxTDvHUz0Xa75qfFFXL3xvEOqf
      -----END RSA PRIVATE KEY-----
    keyPrefix: "/signaling/proxy/server"

  etcd:
    endpoints:
      - https://etcd-01.domain.tld
      - https://etcd-02.domain.tld
    clientCertificate: |
      -----BEGIN CERTIFICATE-----
      MIIFjDCCA3SgAwIBAgIUYXQcZ0dkOOQsIpcpAnWvYPPUtAAwDQYJKoZIhvcNAQEL
      ...snip...
      TGAALAY5lWlwl3BEsn3KxuV72iUpllbWPj6rjicPBk8=
      -----END CERTIFICATE-----
    clientKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAuV0epw6IvrGpVJxYNqnvlcrLpxSHMwoaXvVPCNvD0jKfi+Zp
      ...snip...
      YW4QGLxpT+kRAPyPSZDz2U+k35cxTDvHUz0Xa75qfFFXL3xvEOqf
      -----END RSA PRIVATE KEY-----
    clientCA: |
      -----BEGIN CERTIFICATE-----
      MIIFjDCCA3SgAwIBAgIUYXQcZ0dkOOQsIpcpAnWvYPPUtAAwDQYJKoZIhvcNAQEL
      ...snip...
      TGAALAY5lWlwl3BEsn3KxuV72iUpllbWPj6rjicPBk8=
      -----END CERTIFICATE-----

Cluster of signaling proxies available to external signaling servers¶

This will create a cluster of signaling proxies that can be used by external signaling servers.

networks:
  public-net: |
    {
      "cniVersion": "0.3.1",
      "type": "ipvlan",
      "capabilities": { "ips": true },
      "master": "br0",
      "mode": "l2",
      "ipam": {
        "type": "static"
      }
    }

proxy:
  country: "DE"

  tokens:
    signaling1: |
      -----BEGIN PUBLIC KEY-----
      MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuV0epw6IvrGpVJxYNqnv
      ...snip...
      FQIDAQAB
      -----END PUBLIC KEY-----
    signaling2: |
      -----BEGIN PUBLIC KEY-----
      MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuV0epw6IvrGpVJxYNqnv
      ...snip...
      FQIDAQAB
      -----END PUBLIC KEY-----

  janus:
    gateways:
      - network:
          name: "public-net"
          publicIP: "10.2.0.100"
          gatewayIP: "10.2.0.1"
        portRange:
          min: 20000
          max: 40000
      - network:
          name: "public-net"
          publicIP: "10.2.0.101"
          gatewayIP: "10.2.0.1"
        portRange:
          min: 20000
          max: 40000
      - network:
          name: "public-net"
          publicIP: "10.2.0.102"
          gatewayIP: "10.2.0.1"
        portRange:
          min: 20000
          max: 40000
    resources:
      cpu:
        request: "8"
        limit: "8"
      memory:
        limit: "512Gi"

  ingresses:
    - host: "proxy1.hpb.domain.tld"
      tlsSecretName: "proxy1-hpb-domain-tld-tls"
      annotations:
        cert-manager.io/issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx
    - host: "proxy2.hpb.domain.tld"
      tlsSecretName: "proxy2-hpb-domain-tld-tls"
      annotations:
        cert-manager.io/issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx
    - host: "proxy3.hpb.domain.tld"
      tlsSecretName: "proxy3-hpb-domain-tld-tls"
      annotations:
        cert-manager.io/issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx

Cluster of signaling proxies available to external signaling servers configured through etcd¶

This will create a cluster of signaling proxies that can be used by external signaling servers. The list of allowed signaling servers is stored in an etcd cluster.

When a new signaling server connection is received, the sent tokenId is added to the list of tokenKeyFormats (parameter %s gets replaced) and the etcd cluster is queried for the public key of the token. If a key is found and matches the private key of the signaling server, the connection will be allowed. Otherwise it gets rejected.

To store the public key of token signaling1 for the example below, you can use one of the following commands:

cat pubkey.pem | etcdctl put /signaling/proxy/tokens-staging/signaling1/public-key
cat pubkey.pem | etcdctl put /signaling/proxy/tokens-prod/signaling1/public-key

networks:
  public-net: |
    {
      "cniVersion": "0.3.1",
      "type": "ipvlan",
      "capabilities": { "ips": true },
      "master": "br0",
      "mode": "l2",
      "ipam": {
        "type": "static"
      }
    }

proxy:
  country: "DE"

  tokenKeyFormats:
    - /signaling/proxy/tokens-staging/%s/public-key
    - /signaling/proxy/tokens-prod/%s/public-key

  etcd:
    endpoints:
      - https://etcd-01.domain.tld
      - https://etcd-02.domain.tld
    clientCertificate: |
      -----BEGIN CERTIFICATE-----
      MIIFjDCCA3SgAwIBAgIUYXQcZ0dkOOQsIpcpAnWvYPPUtAAwDQYJKoZIhvcNAQEL
      ...snip...
      TGAALAY5lWlwl3BEsn3KxuV72iUpllbWPj6rjicPBk8=
      -----END CERTIFICATE-----
    clientKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAuV0epw6IvrGpVJxYNqnvlcrLpxSHMwoaXvVPCNvD0jKfi+Zp
      ...snip...
      YW4QGLxpT+kRAPyPSZDz2U+k35cxTDvHUz0Xa75qfFFXL3xvEOqf
      -----END RSA PRIVATE KEY-----
    clientCA: |
      -----BEGIN CERTIFICATE-----
      MIIFjDCCA3SgAwIBAgIUYXQcZ0dkOOQsIpcpAnWvYPPUtAAwDQYJKoZIhvcNAQEL
      ...snip...
      TGAALAY5lWlwl3BEsn3KxuV72iUpllbWPj6rjicPBk8=
      -----END CERTIFICATE-----

  janus:
    gateways:
      - network:
          name: "public-net"
          publicIP: "10.2.0.100"
          gatewayIP: "10.2.0.1"
        portRange:
          min: 20000
          max: 40000
      - network:
          name: "public-net"
          publicIP: "10.2.0.101"
          gatewayIP: "10.2.0.1"
        portRange:
          min: 20000
          max: 40000
      - network:
          name: "public-net"
          publicIP: "10.2.0.102"
          gatewayIP: "10.2.0.1"
        portRange:
          min: 20000
          max: 40000
    resources:
      cpu:
        request: "8"
        limit: "8"
      memory:
        limit: "512Gi"

  ingresses:
    - host: "proxy1.hpb.domain.tld"
      tlsSecretName: "proxy1-hpb-domain-tld-tls"
      annotations:
        cert-manager.io/issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx
    - host: "proxy2.hpb.domain.tld"
      tlsSecretName: "proxy2-hpb-domain-tld-tls"
      annotations:
        cert-manager.io/issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx
    - host: "proxy3.hpb.domain.tld"
      tlsSecretName: "proxy3-hpb-domain-tld-tls"
      annotations:
        cert-manager.io/issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx

Running ingresses behind F5 NGINX Ingress Controller¶

If the signaling server or -proxy are deployed in a cluster where the ingress is handled by the F5 NGINX Ingress Controller, an additional annotation nginx.org/websocket-services must be added to the ingress configurations, so the WebSocket requests are forwarded correctly.

Forther information is available at https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#backend-services-upstreams

Signaling server¶

signaling:

  # Other configuration

  ingress:
    host: "signaling.domain.tld"
    tlsSecretName: "signaling-domain-tld-tls"
    annotations:
      cert-manager.io/issuer: letsencrypt-prod
      # Other annotations
      nginx.org/websocket-services: "service-of-signaling-server"

The name of the service for the signaling server is <cluster-name-from-helm>-signaling-server.

For helm install mysignaling hpb/cluster --values mysignaling.yaml the name will be mysignaling-signaling-server.

Signaling proxy¶

proxy:

  # Other configuration

  ingresses:
    - host: "proxy.hpb.domain.tld"
      tlsSecretName: "proxy-hpb-domain-tld-tls"
      annotations:
        cert-manager.io/issuer: letsencrypt-prod
        # Other annotations
        nginx.org/websocket-services: "service-of-signaling-proxy"

The name of the service for the signaling proxy is <cluster-name-from-helm>-signaling-proxy.

For helm install myproxy hpb/cluster --values myproxy.yaml the name will be myproxy-signaling-proxy.

Upgrading¶

See the Helm documentation for details on upgrades and rollbacks at https://helm.sh/docs/intro/using_helm/#helm-upgrade-and-helm-rollback-upgrading-a-release-and-recovering-on-failure

Please note that upgrading the controller might update all previously created HPB clusters (e.g. if one of the underlying images has changed).

Uninstall¶

Once a HPB cluster is no longer needed, it can be uninstalled with Helm:

$ helm uninstall cluster1

The controller will then remove the deployments for the signaling servers and/or proxies.

More details on uninstalling are available in the Helm documentation at https://helm.sh/docs/intro/using_helm/#helm-uninstall-uninstalling-a-release