Scale agent deployments

Scale an AgentRuntime to a fixed replica count, or let an HPA or KEDA adjust it with load.

Prerequisites

A running AgentRuntime. The examples call it my-agent.
For HPA, the Kubernetes Metrics Server.
For KEDA, KEDA installed in the cluster, and Prometheus if you scale on connections. See Autoscaling for how the operator wires both.

Set a fixed replica count

Set spec.runtime.replicas:

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: AgentRuntime
metadata:
  name: my-agent
spec:
  runtime:
    replicas: 3
  # ...

Or patch it:

kubectl patch agentruntime my-agent --type=merge \
  -p '{"spec":{"runtime":{"replicas":5}}}'

When autoscaling is enabled, the autoscaler owns the replica count and runtime.replicas has no effect.

Choose an autoscaler

Use HPA when CPU and memory track your load and you want no extra dependencies. Use KEDA for connection-based scaling, cron schedules, external events or scale-to-zero.

Keep minReplicas: 1 for interactive agents. A scaled-to-zero agent refuses the first connection, and the default connections trigger cannot wake it, so the API server rejects a KEDA policy with minReplicas: 0 and no keda.triggers. To scale to zero, list a trigger that does not depend on the agent’s pods, such as cron.

Autoscale with HPA

Enable the HPA:

spec:
  runtime:
    autoscaling:
      enabled: true
      type: hpa
      minReplicas: 2
      maxReplicas: 10
      targetMemoryUtilizationPercentage: 70
      targetCPUUtilizationPercentage: 90

minReplicas defaults to 1, maxReplicas to 100, the memory target to 70 and the CPU target to 90. The CPU target cannot be turned off.

HPA utilisation targets are percentages of the container’s resource requests. See Set resource requests.

Autoscale with KEDA

Set type: keda:

spec:
  runtime:
    autoscaling:
      enabled: true
      type: keda
      minReplicas: 1
      maxReplicas: 20

With no triggers, the operator adds a Prometheus trigger on omnia_agent_connections_active for the agent, with a threshold of keda.connectionThreshold connections per pod (default 1000, half the facade’s per-pod cap of 2000). The trigger queries the Prometheus server that the Omnia chart installs. Lower connectionThreshold for audio and media agents.

If your Prometheus runs elsewhere, or you want other signals, define keda.triggers yourself. The serverAddress below is illustrative:

spec:
  runtime:
    autoscaling:
      enabled: true
      type: keda
      minReplicas: 1
      maxReplicas: 20
      keda:
        pollingInterval: 30
        cooldownPeriod: 300
        triggers:
          - type: prometheus
            metadata:
              serverAddress: "http://prometheus:9090"
              query: 'sum(omnia_agent_connections_active{agent="my-agent",namespace="default"}) or vector(0)'
              threshold: "10"

The or vector(0) makes the query return 0 when no series exist.

A cron trigger raises the replica count during business hours:

triggers:
  - type: cron
    metadata:
      timezone: "America/New_York"
      start: "0 8 * * 1-5"   # 8am Mon-Fri
      end: "0 18 * * 1-5"    # 6pm Mon-Fri
      desiredReplicas: "5"

Metrics to scale on, all labelled agent and namespace:

Metric	Meaning	Scale on it
`omnia_agent_connections_active`	Current WebSocket connections	Yes, a direct load indicator
`omnia_agent_requests_inflight`	Requests being processed (pending LLM calls)	Yes, shows processing load
`omnia_agent_sessions_active`	Active sessions	Depends on session lifetime
`omnia_agent_request_duration_seconds`	Request latency	No, it trails the load

See Set Up Observability for the full metric list.

Tune autoscaling

For a latency-sensitive agent, lower the memory target and keep pods longer (HPA):

autoscaling:
  type: hpa
  targetMemoryUtilizationPercentage: 50
  scaleDownStabilizationSeconds: 600

For a cost-sensitive agent, allow more connections per pod (KEDA):

autoscaling:
  type: keda
  minReplicas: 1
  keda:
    connectionThreshold: 400

For predictable traffic, combine a cron floor with connection-based scaling (KEDA). Add the Prometheus trigger as well, because defining any trigger replaces the default one:

autoscaling:
  type: keda
  keda:
    triggers:
      - type: cron
        metadata:
          timezone: "UTC"
          start: "0 8 * * 1-5"
          end: "0 18 * * 1-5"
          desiredReplicas: "5"
      - type: prometheus
        metadata:
          serverAddress: "http://prometheus:9090"
          query: 'sum(omnia_agent_connections_active{agent="my-agent",namespace="default"}) or vector(0)'
          threshold: "10"

Set resource requests

Every container in the agent pod has CPU and memory requests, so an HPA utilisation target can compute. The operator sets these defaults, as requests with no limits:

Container	CPU	Memory
`facade`	`100m`	`128Mi`
`runtime`	`100m`	`256Mi`
`policy-broker`	`50m`	`64Mi`

On a small cluster running many mostly idle agents, scale every default CPU request at once with the operator flag --agent-cpu-request-percent (default 100). For example, set operator.extraArgs: ["--agent-cpu-request-percent=10"] in the chart values. Every container still gets a request, so an HPA can still compute utilisation, but utilisation is then measured against the smaller request.

spec.runtime.resources replaces the runtime container’s defaults. The facade and policy-broker defaults cannot be changed per agent:

spec:
  runtime:
    resources:
      requests:
        cpu: "500m"
        memory: "256Mi"
      limits:
        cpu: "1000m"
        memory: "512Mi"

Run more than one replica

Use the Redis context store so any replica can resume a conversation:

spec:
  context:
    type: redis
    storeRef:
      name: redis-credentials

storeRef names a Secret with a url key. The operator owns the agent’s Service and resets its spec on every reconcile, so you cannot set sessionAffinity on it. See Sessions for how the memory store behaves with several replicas.

Verify

Check replica status. The Ready column shows ready replicas:

kubectl get agentruntime my-agent
kubectl describe agentruntime my-agent

The AutoscalingReady condition reports whether the HPA or ScaledObject was reconciled. For HPA, inspect the HPA, which is named after the agent:

kubectl get hpa my-agent
kubectl describe hpa my-agent

For KEDA, inspect the ScaledObject, which is also named after the agent, and the HPA that KEDA manages:

kubectl get scaledobject my-agent
kubectl get hpa keda-hpa-my-agent

If AutoscalingReady is False with reason KEDANotInstalled, install KEDA. The agent stays at its static replica count until you do.

Autoscaling explains HPA versus KEDA and what scale-to-zero costs.
Set Up Observability covers the scaling metrics.
AgentRuntime reference lists every autoscaling field.