Scaling Context Chat using Kubernetes¶

Starting with version 5.4.0 of context chat Kubernetes is supported for deployment of the backend to scale ContextChat to large instances.

For Kubernetes deployment the app is divided into three kubernetes pod roles:

• updatesproc Which will process meta data updates for indexed items
• requestproc Which will process user requests for the Context Chat feature
• indexing Which will take care of the initial indexing of documents and the subsequent indexing of new documents and changes to existing documents

You can scale these roles independently using KEDA, e.g. with the following example configurations. (Note that you may need to tune the values for your specific setup.)

When running Context Chat in Kubernetes, you MUST use an external vector Db and it usually also makes sense to set up an external embedding LLM service (see https://docs.nextcloud.com/server/latest/admin_manual/ai/app_context_chat.html#scaling for details)

Update processing¶

    apiVersion: keda.sh/v1alpha1
    kind: ScaledObject
    metadata:
      name: ccb-scaler-updatesproc
      namespace: <your-nextcloud-exapp-namespace>
    spec:
      scaleTargetRef:
        name: nc-app-context-chat-backend-updatesproc
      pollingInterval: 15
      cooldownPeriod: 120
      initialCooldownPeriod: 60
      minReplicaCount: 1
      maxReplicaCount: 4
      triggers:
        - type: metrics-api
          metadata:
            url: "https://<your-nextcloud-host>/ocs/v2.php/apps/context_chat/queues/actions/stats?format=json"
            valueLocation: "ocs.data.scheduled"
            targetValue: "5"
            authMode: "basic"
          authenticationRef:
            name: <your-authentication-secret-name>

Request processing¶

    apiVersion: keda.sh/v1alpha1
    kind: ScaledObject
    metadata:
      name: ccb-scaler-requestproc
      namespace: <your-nextcloud-exapp-namespace>
    spec:
      scaleTargetRef:
        name: nc-app-context-chat-backend-requestproc
      pollingInterval: 15
      cooldownPeriod: 120
      initialCooldownPeriod: 60
      minReplicaCount: 1
      maxReplicaCount: 4
      triggers:
        - type: metrics-api
          metadata:
            url: "https://<your-nextcloud-host>/ocs/v2.php/taskprocessing/queue_stats?format=json&taskTypeId=context_chat:context_chat,context_chat:context_chat_search"
            valueLocation: "ocs.data.scheduled_count"
            targetValue: "5"
            authMode: "basic"
          authenticationRef:
            name: <your-authentication-secret-name>

Indexing¶

    apiVersion: keda.sh/v1alpha1
    kind: ScaledObject
    metadata:
      name: ccb-scaler-indexing
      namespace: <your-nextcloud-exapp-namespace>
    spec:
      scaleTargetRef:
        name: nc-app-context-chat-backend-indexing
      pollingInterval: 15
      cooldownPeriod: 120
      initialCooldownPeriod: 60
      minReplicaCount: 1
      maxReplicaCount: 4
      triggers:
        - type: metrics-api
          metadata:
            url: "https://<your-nextcloud-host>/ocs/v2.php/apps/context_chat/queues/documents/stats?format=json"
            valueLocation: "ocs.data.scheduled"
            targetValue: "5"
            authMode: "basic"
          authenticationRef:
            name: <your-authentication-secret-name>