Skip to content
ALTAIRA LABS
DocsBlog
OmniaOmnia

Autoscaling

Omnia offers two autoscaling mechanisms for an AgentRuntime: the standard Kubernetes HorizontalPodAutoscaler (HPA) and KEDA. This page explains how the operator wires each one up, why the HPA defaults look the way they do, and what scale-to-zero costs an interactive agent. To pick one, see Scale agent deployments.

How the operator wires autoscaling

Section titled “How the operator wires autoscaling”

The operator resolves one autoscaling policy per agent. If the agent sets spec.runtime.autoscaling, that block applies as a whole. Otherwise the agent inherits the policy on its Workspace service group (services[].autoscaling), and if neither exists the agent runs at static replicas. The operator then creates an HPA or a KEDA ScaledObject that targets the agent’s Deployment, and deletes the other kind if the agent switched type.

flowchart LR
    AR[AgentRuntime spec.runtime.autoscaling] --> OP[Operator AgentRuntimeReconciler]
    WS[Workspace services autoscaling] --> OP
    OP -->|type hpa| HPA[HorizontalPodAutoscaler]
    OP -->|type keda| SO[KEDA ScaledObject]
    SO --> KEDA[KEDA]
    KEDA -->|polls| PROM[Prometheus]
    PROM -->|scrapes| POD[Agent pod facade]
    KEDA -->|manages| HPA2[HorizontalPodAutoscaler]
    HPA --> DEP[Agent Deployment]
    HPA2 --> DEP

The Workspace default exists because agents created through the dashboard or a deploy tool cannot easily express per-agent scaling. A workspace owner turns autoscaling on for every agent in a group in one place. The agent’s own block replaces the default entirely instead of merging with it, so a partial block never silently combines with settings you cannot see. See the Workspace reference.

If type: keda is requested and the KEDA CRDs are absent, the operator does not fail the reconcile. It sets AutoscalingReady=False with reason KEDANotInstalled and leaves the agent at static replicas.

Why agents scale differently from web services

Section titled “Why agents scale differently from web services”

Agents have a load profile that CPU-based scaling reads poorly:

  • They are I/O bound. Most of a request is spent waiting on an LLM response, so CPU stays low while the agent is busy.
  • WebSocket connections last for a whole conversation, so load is connections held open rather than requests per second.
  • Each connection and session holds context in memory.
  • LLM latency varies from one second to well over thirty.

The defaults follow from this profile.

HPA: resource-based scaling

Section titled “HPA: resource-based scaling”

An HPA reads CPU and memory utilisation from the Kubernetes Metrics Server, compares it with a target, and adjusts the Deployment’s replica count. It needs nothing beyond the Metrics Server.

Omnia builds the HPA with two metrics. Memory utilisation is the primary one, with a default target of 70%, because every connection and its session context consume memory while the CPU stays idle. CPU utilisation is the secondary one, with a default target of 90%.

The operator also sets the HPA’s scaling behaviour. Scale-up has no stabilisation window and can double the pod count or add four pods every 15 seconds, whichever is larger. Scale-down waits 300 seconds by default and removes at most 50% of pods per minute. Under load, a slow scale-up means dropped or delayed conversations. Scale-down can be slow, because WebSocket clients tend to reconnect shortly after a drop, and a pod that was needed a minute ago may be needed again.

The cost of memory-based scaling is that memory is a lagging and sticky signal. A pod’s memory usage does not fall the moment its connections close, so HPA scales down later than the load would justify. For most agents that is the right side to err on.

KEDA: custom metrics, cron and scale-to-zero

Section titled “KEDA: custom metrics, cron and scale-to-zero”

KEDA extends the HPA model. It polls external sources such as Prometheus or a queue, manages an HPA on your behalf, and can take a Deployment to zero replicas and bring it back. If KEDA is already installed in the cluster, keep keda.enabled=false in the Omnia chart and the existing installation works with Omnia’s ScaledObject resources.

With no custom triggers, Omnia adds a Prometheus trigger that sums omnia_agent_connections_active for the agent and compares it with a per-pod connectionThreshold, which defaults to 1000: half the facade’s cap of 2000 connections per pod, so a pod scales out with room left to absorb a surge or a lost replica. That default targets text workloads. Audio and media sessions are far heavier per connection, so lower the threshold for them. The trigger points at the Prometheus server installed by the Omnia chart. If your Prometheus lives elsewhere, define the trigger explicitly.

Connections are a better scaling signal than memory because they measure demand directly and move as soon as a client connects or leaves. The price is a dependency on Prometheus, which is an optional chart component. Cron triggers cover predictable traffic such as business hours, and they combine with the Prometheus trigger so that cron sets a floor and connections handle spikes. See Scale agent deployments for trigger examples, and the AgentRuntime reference for the fields.

What scale-to-zero costs

Section titled “What scale-to-zero costs”

Setting minReplicas: 0 on a KEDA policy lets an idle agent cost nothing. For an interactive WebSocket agent the saving comes with two problems.

First, the first client after an idle period finds no pod to talk to. The connection fails, and the client has to retry once a pod is ready, which takes as long as scheduling, image pull and startup.

Second, the default connections trigger cannot wake the agent. The omnia_agent_connections_active gauge is exported by the agent pod itself, so with zero pods the query returns 0 whatever clients attempt. Something that does not depend on the pod has to raise the replica count, such as a cron trigger or an external event source. The API server therefore rejects minReplicas: 0 on a KEDA policy that lists no keda.triggers. It cannot tell whether a trigger you list depends on the pods, so a Prometheus trigger on the agent’s own metrics has the same problem.

sequenceDiagram
    participant C as Client
    participant S as Service
    participant K as KEDA
    participant P as Agent pod


    Note over P: Scaled to 0
    C->>S: WebSocket connect
    S-->>C: Connection fails, no endpoints
    Note over K: Cron trigger fires
    K->>P: Scale Deployment to 1
    Note over P: Pod starting, then ready
    C->>S: Retry connect
    S->>P: Forward
    P-->>C: Connected

Scale-to-zero suits non-interactive agents and workloads that tolerate an occasional failed first connection. For interactive agents, minReplicas: 1 keeps the benefits of scaling from one pod to many, and the cost of one idle pod is usually smaller than the cost of a failed first conversation. The KEDA HTTP add-on, which buffers requests during scale-up, is not part of Omnia’s wiring.

Choosing between them

Section titled “Choosing between them”

HPA is the simple choice. It has no extra dependencies, uses signals every cluster has, and is enough when memory and CPU track your load. KEDA earns its extra moving parts when you want to scale on connections or queue depth, schedule capacity ahead of known traffic, or scale to zero. Many teams begin with HPA and move to KEDA only when memory stops being a good proxy for load.

Section titled “Related pages”