Rollout strategies

Progressive rollouts deploy a change to an AI agent to a slice of traffic first, and widen it only while the change holds up. This page explains the design behind Omnia’s rollout system and the trade-offs in each routing mode.

Why progressive rollouts for AI agents?

Traditional application rollouts focus on binary correctness: the new version either works or it crashes. AI agents have a subtler failure mode, quality degradation. A prompt change might deploy successfully but produce worse responses, hallucinate more, or misuse tools. These failures don’t trigger health checks.

Progressive rollouts address this by:

Exposing the new version to a small percentage of traffic first
Running automated quality analysis before increasing exposure
Rolling back automatically when analysis detects degradation
Supporting A/B experiments with a fixed traffic split

The dual deployment model

When a rollout is active, the controller creates a second Deployment, <agent>-canary, beside the stable one. Both sit behind the agent’s Service, which selects the pods of both.

The diagram shows the mesh routing mode: the pods carry a track label, and an Istio VirtualService and DestinationRule split traffic by that label.

graph TB
    VS[VirtualService: my-agent-rollout] -->|weight split| DR[DestinationRule: my-agent-rollout]
    DR -->|subset: stable<br/>track=stable| STABLE[Deployment my-agent<br/>Current spec]
    DR -->|subset: canary<br/>track=canary| CANDIDATE[Deployment my-agent-canary<br/>Spec + overrides]
    CM[ConfigMap my-agent-canary-config<br/>candidate provider refs] -->|mounted| CANDIDATE
    SVC[Service: my-agent] -.-> STABLE
    SVC -.-> CANDIDATE

Both Deployments run the same agent image. The only differences come from the rollout.candidate overrides: typically a different PromptPack, provider list, or tool registry. The pods of the two Deployments carry the label omnia.altairalabs.ai/track with the value stable or canary.

How the weight reaches the pods depends on trafficRouting.mode:

mesh: the controller creates and owns a VirtualService and a DestinationRule, both named <agent>-rollout, and updates the weights on each step. The DestinationRule subsets select on the track label and are named stable and canary unless you override them.
replicaWeighted: no mesh is involved. The controller splits the replica count between the two Deployments, so the Service spreads requests across pods in roughly that proportion.
external: you author the VirtualService and DestinationRule, and the controller patches the weights on them.

When you leave mode unset, the controller picks mesh if the mesh is available and replicaWeighted otherwise. A config that sets istio without mode resolves to external. A request for mesh without an available mesh falls back to replicaWeighted and sets the TrafficRouting condition to False with reason Degraded. The controller resolves a mode only when the rollout has a trafficRouting block (an empty one is enough). Without the block it applies no weights.

The candidate’s pods receive their own resolved configuration: the operator writes the candidate’s provider refs into a per-agent override ConfigMap mounted into the candidate pods, so the runtime resolves the candidate’s providers, not the stable spec’s. A canary therefore exercises the overridden behaviour rather than silently running stable’s config.

The stable Deployment serves throughout the rollout, and promotion is zero-downtime too (see Promotion and rollback). Both versions share the same Service, DNS, and TLS. After promotion or rollback, the candidate Deployment and its override ConfigMap are deleted.

Choosing a routing mode

mesh and external deliver the weight you ask for, because the mesh splits requests. Both need Istio. In mesh the controller owns the routing objects, so you author nothing. external suits a platform team that already manages its own VirtualServices.

replicaWeighted needs no mesh and costs accuracy. The weight you get is the ratio of candidate pods to total pods. The controller splits spec.runtime.replicas between the two Deployments and rounds, so four replicas express 0, 25, 50, 75 and 100. With fewer than two replicas the split is binary: a weight of 50 or more sends everything to the candidate and anything lower sends nothing. status.rollout.trafficWeightEnforced is false in this mode so dashboards can label the weight as approximate.

Candidate overrides

The candidate is a sparse override, not a full copy of the spec. You set only the fields you want to change: the PromptPack reference, the provider list, or the ToolRegistry reference. Anything you leave out inherits from the main spec. This keeps rollout configurations small and avoids drift between the two versions.

A rollout can also start on its own: with rollout.trigger, the controller sets a newer PromptPack version from a release channel as the candidate, and the steps canary it. See the AgentRuntime reference for every candidate field and an example.

Step-based progression

Rollouts are defined as an ordered sequence of steps. Each step is one of three types:

setWeight adjusts the candidate’s traffic percentage (0-100).
pause waits for a duration, or indefinitely until manually resumed.
analysis runs a RolloutAnalysis template and evaluates the result.

The controller processes steps sequentially. It only advances to the next step when the current one completes successfully. A failed analysis step triggers the configured rollback behavior.

Canary pattern

Gradually increase traffic, pausing between steps for observation:

steps:
  - setWeight: 10
  - pause: { duration: "5m" }
  - setWeight: 50
  - pause: { duration: "10m" }
  - setWeight: 100

Blue/green pattern

Run analysis on the candidate with zero traffic, then flip all at once. The leading setWeight: 0 matters: the controller applies no weight during an analysis step, so without it the Service spreads requests across both Deployments:

steps:
  - setWeight: 0
  - analysis: { templateName: smoke-test }
  - setWeight: 100

Zero-traffic analysis needs mesh or external routing. In replicaWeighted mode, weight 0 scales the candidate to zero pods.

Experiment pattern

Split traffic evenly for A/B testing, and pin each user to a variant with sticky sessions in external mode:

steps:
  - setWeight: 50
  - pause: {}  # Indefinite, manual promotion

All three patterns use the same step primitives. The strategy emerges from how you compose the steps.

Traffic routing with Istio

In mesh mode the controller owns two Istio resources per agent, both named <agent>-rollout and owned by the AgentRuntime, so deleting the AgentRuntime deletes them:

VirtualService: one rollout route with a weighted destination for each subset. It matches the agent’s Service DNS name unless you set trafficRouting.mesh.hosts.
DestinationRule: two subsets that select pods by the omnia.altairalabs.ai/track label (stable and canary).

Edits you make to these objects are overwritten on the next reconcile. In an Istio ambient mesh, the ztunnel alone is L4-only and bypasses the VirtualService. Set trafficRouting.mesh.waypoint and the controller labels the agent Service istio.io/use-waypoint=<waypoint>, so the waypoint performs the weighted split.

In external mode you create the VirtualService and DestinationRule yourself and the controller patches weights on them. It only modifies the routes listed in trafficRouting.istio.virtualService.routes and leaves other routes untouched. The resources must exist before the rollout starts: a missing VirtualService is reported as a degraded TrafficRouting condition and no weights are applied.

Cohort tracking

The facade tags each session with the rollout variant and cohort of the request that opened it. Three sources feed the tags:

The x-omnia-variant request header, with the value stable or candidate.
The OMNIA_VARIANT environment variable, which is stable on stable pods and candidate on candidate pods.
The x-omnia-cohort-id request header, which carries any caller-supplied cohort identifier.

In mesh mode the owned VirtualService sets x-omnia-variant on each weighted destination, so the value follows the route taken. The facade prefers the header and falls back to OMNIA_VARIANT. In replicaWeighted mode nothing sets the header, so the pod’s environment variable decides. The variant value is candidate even though the subset is named canary.

The controller never sets x-omnia-cohort-id. The facade reads it from the request and has no fallback, so it is empty unless something upstream sets it. Both values are stored on the session (variant and cohort_id) and recorded as the omnia.variant and omnia.cohort.id attributes on facade message spans. Because the facade accepts both headers from any caller, they identify a cohort for analysis and are not an access control.

Sticky sessions

For experiments that require consistent user routing, configure stickySession:

rollout:
  stickySession:
    hashOn: "x-user-id"

The controller sets a consistent hash on the header you name in the DestinationRule that you reference under trafficRouting.istio, so the same user always reaches the same variant. It removes the hash when the rollout ends. Sticky sessions need that istio reference. The DestinationRule the controller owns in mesh mode never gets a hash, and replicaWeighted has no routing layer, so neither mode applies stickySession. Each request is then routed independently by weight, and a user might see both variants across consecutive messages.

Promotion and rollback

A rollout ends in one of three states.

Idle: when rollout.candidate matches the current spec (or is absent), no rollout is active. The controller maintains a single Deployment.

Promotion: when the rollout completes all steps, the candidate’s config is merged into the main spec and the stable Deployment rolls to it in the background. Throughout that roll, 100% of traffic stays on the already-validated, warm candidate. Only once the stable Deployment is fully rolled out and healthy on the new config does traffic cut back to it and the candidate Deployment get deleted. No request is ever served from a cold or restarting pod, so promotion is zero-downtime. While the background roll is underway, status.rollout.promoting is true and the RolloutActive condition reports reason Promoting.

Rollback: the candidate is reverted to match the current spec, and the candidate Deployment is deleted. Traffic returns to 100% stable.

Rollback behavior is controlled by rollout.rollback.mode:

automatic rolls back when an analysis step fails or the candidate Deployment exceeds its progress deadline.
manual (the default) never rolls back on its own. A failed analysis leaves the rollout on its step, and you roll back by editing rollout.candidate to match the current spec.
disabled behaves like manual.

With manual, a failed analysis leaves the rollout on its current step with RolloutActive=True, reason AnalysisFailed, and a warning event. The controller sets no timed requeue, but the rollout is not idle: the status write that records the failure triggers another reconcile, which re-runs the analysis from a fresh measurement sequence, so a metric that has recovered can pass and the rollout continues. The re-runs stop only when one fails on its first pass with the same message, because the status write is then unchanged and fires no event. The rollout then waits for the next event, such as an edit to the AgentRuntime. Nothing in the controller reads disabled separately, so a failed analysis leaves the rollout on its step exactly as it does under manual. The pod-health rollback runs only under automatic.

Analysis integration

Analysis steps run a RolloutAnalysis CRD, a reusable template whose metrics come from the prometheus provider, the web provider (an HTTP endpoint) or the arenaEval provider (an evaluation score). A Prometheus metric looks like this:

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: RolloutAnalysis
metadata:
  name: quality-check
spec:
  metrics:
    - name: error-rate
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(omnia_agent_requests_total{status="error", variant="candidate"}[5m]))
            /
            sum(rate(omnia_agent_requests_total{variant="candidate"}[5m]))
      successCondition: "result[0] < 0.05"
      count: 5
      failureLimit: 3
      interval: "1m"

The controller takes count measurements of each metric, spaced interval apart, and evaluates successCondition against each one:

The analysis step passes (and the rollout advances) only once every metric has completed all count measurements without exceeding its failureLimit.
The analysis step fails as soon as any metric’s failed measurements exceed its failureLimit, which triggers the configured rollback behavior. With the default failureLimit: 0, the first failed measurement is decisive.
A measurement that cannot be taken at all (Prometheus unreachable, template missing, malformed condition) is an error, not a failure. The controller retries every 30s and never rolls back on it, because the fault is in the observation path rather than in the candidate.

Measurement progress is recorded in the AgentRuntime’s status.rollout.analysis and survives controller restarts. It is keyed on the step, the template, the template’s spec revision and the candidate version. A change to any of them discards the progress and starts a fresh sequence, so a measurement taken against one candidate never counts against another.

An arenaEval metric reads the average score of one evaluation definition for the candidate’s sessions. Until the evaluation has produced a result for the candidate, the measurement is an error and retries every 30s, so a young candidate is never rolled back for having no score yet.

Progressive Rollouts Tutorial: step-by-step walkthrough using replicaWeighted routing
AgentRuntime CRD Reference: field-by-field specification
RolloutAnalysis CRD Reference: analysis template fields