# Rollout strategies

Progressive rollouts deploy a change to an AI agent to a slice of traffic first, and widen it only while the change holds up. This page explains the design behind Omnia's rollout system and the trade-offs in each routing mode.

## Why progressive rollouts for AI agents?

Traditional application rollouts focus on binary correctness: the new version either works or it crashes. AI agents have a subtler failure mode, **quality degradation**. A prompt change might deploy successfully but produce worse responses, hallucinate more, or misuse tools. These failures don't trigger health checks.

Progressive rollouts address this by:

- Exposing the new version to a small percentage of traffic first
- Running automated quality analysis before increasing exposure
- Rolling back automatically when analysis detects degradation
- Supporting A/B experiments with a fixed traffic split

## The dual deployment model

When a rollout is active, the controller creates a second Deployment, `<agent>-canary`, beside the stable one. Both sit behind the agent's Service, which selects the pods of both.

The diagram shows the `mesh` routing mode: the pods carry a `track` label, and an Istio VirtualService and DestinationRule split traffic by that label.

```mermaid
graph TB
    VS[VirtualService: my-agent-rollout] -->|weight split| DR[DestinationRule: my-agent-rollout]
    DR -->|subset: stable<br/>track=stable| STABLE[Deployment my-agent<br/>Current spec]
    DR -->|subset: canary<br/>track=canary| CANDIDATE[Deployment my-agent-canary<br/>Spec + overrides]
    CM[ConfigMap my-agent-canary-config<br/>candidate provider refs] -->|mounted| CANDIDATE
    SVC[Service: my-agent] -.-> STABLE
    SVC -.-> CANDIDATE
```

Both Deployments run the same agent image. The only differences come from the `rollout.candidate` overrides: typically a different PromptPack, provider list, or tool registry. The pods of the two Deployments carry the label `omnia.altairalabs.ai/track` with the value `stable` or `canary`.

How the weight reaches the pods depends on `trafficRouting.mode`:

- `mesh`: the controller creates and owns a VirtualService and a DestinationRule, both named `<agent>-rollout`, and updates the weights on each step. The DestinationRule subsets select on the `track` label and are named `stable` and `canary` unless you override them.
- `replicaWeighted`: no mesh is involved. The controller splits the replica count between the two Deployments, so the Service spreads requests across pods in roughly that proportion.
- `external`: you author the VirtualService and DestinationRule, and the controller patches the weights on them.

When you leave `mode` unset, the controller picks `mesh` if the mesh is available and `replicaWeighted` otherwise. A config that sets `istio` without `mode` resolves to `external`. A request for `mesh` without an available mesh falls back to `replicaWeighted` and sets the `TrafficRouting` condition to `False` with reason `Degraded`. The controller resolves a mode only when the rollout has a `trafficRouting` block (an empty one is enough). Without the block it applies no weights.

The candidate's pods receive their **own** resolved configuration: the operator writes the candidate's provider refs into a per-agent override ConfigMap mounted into the candidate pods, so the runtime resolves the candidate's providers, not the stable spec's. A canary therefore exercises the overridden behaviour rather than silently running stable's config.

The stable Deployment serves throughout the rollout, and promotion is zero-downtime too (see [Promotion and rollback](#promotion-and-rollback)). Both versions share the same Service, DNS, and TLS. After promotion or rollback, the candidate Deployment and its override ConfigMap are deleted.

### Choosing a routing mode

`mesh` and `external` deliver the weight you ask for, because the mesh splits requests. Both need Istio. In `mesh` the controller owns the routing objects, so you author nothing. `external` suits a platform team that already manages its own VirtualServices.

`replicaWeighted` needs no mesh and costs accuracy. The weight you get is the ratio of candidate pods to total pods. The controller splits `spec.runtime.replicas` between the two Deployments and rounds, so four replicas express 0, 25, 50, 75 and 100. With fewer than two replicas the split is binary: a weight of 50 or more sends everything to the candidate and anything lower sends nothing. `status.rollout.trafficWeightEnforced` is `false` in this mode so dashboards can label the weight as approximate.

## Candidate overrides

The candidate is a **sparse override**, not a full copy of the spec. You set only the fields you want to change: the PromptPack reference, the provider list, or the ToolRegistry reference. Anything you leave out inherits from the main spec. This keeps rollout configurations small and avoids drift between the two versions.

A rollout can also start on its own: with `rollout.trigger`, the controller sets a newer PromptPack version from a release channel as the candidate, and the steps canary it. See the [AgentRuntime reference](/reference/core/agentruntime/#rollout) for every candidate field and an example.

## Step-based progression

Rollouts are defined as an ordered sequence of steps. Each step is one of three types:

- `setWeight` adjusts the candidate's traffic percentage (0-100).
- `pause` waits for a duration, or indefinitely until manually resumed.
- `analysis` runs a RolloutAnalysis template and evaluates the result.

The controller processes steps sequentially. It only advances to the next step when the current one completes successfully. A failed analysis step triggers the configured rollback behavior.

### Canary pattern

Gradually increase traffic, pausing between steps for observation:

```yaml
steps:
  - setWeight: 10
  - pause: { duration: "5m" }
  - setWeight: 50
  - pause: { duration: "10m" }
  - setWeight: 100
```

### Blue/green pattern

Run analysis on the candidate with zero traffic, then flip all at once. The leading `setWeight: 0` matters: the controller applies no weight during an `analysis` step, so without it the Service spreads requests across both Deployments:

```yaml
steps:
  - setWeight: 0
  - analysis: { templateName: smoke-test }
  - setWeight: 100
```

Zero-traffic analysis needs `mesh` or `external` routing. In `replicaWeighted` mode, weight 0 scales the candidate to zero pods.

### Experiment pattern

Split traffic evenly for A/B testing, and pin each user to a variant with sticky sessions in `external` mode:

```yaml
steps:
  - setWeight: 50
  - pause: {}  # Indefinite, manual promotion
```

All three patterns use the same step primitives. The strategy emerges from how you compose the steps.

## Traffic routing with Istio

In `mesh` mode the controller owns two Istio resources per agent, both named `<agent>-rollout` and owned by the AgentRuntime, so deleting the AgentRuntime deletes them:

- **VirtualService**: one `rollout` route with a weighted destination for each subset. It matches the agent's Service DNS name unless you set `trafficRouting.mesh.hosts`.
- **DestinationRule**: two subsets that select pods by the `omnia.altairalabs.ai/track` label (`stable` and `canary`).

Edits you make to these objects are overwritten on the next reconcile. In an Istio ambient mesh, the ztunnel alone is L4-only and bypasses the VirtualService. Set `trafficRouting.mesh.waypoint` and the controller labels the agent Service `istio.io/use-waypoint=<waypoint>`, so the waypoint performs the weighted split.

In `external` mode you create the VirtualService and DestinationRule yourself and the controller patches weights on them. It only modifies the routes listed in `trafficRouting.istio.virtualService.routes` and leaves other routes untouched. The resources must exist before the rollout starts: a missing VirtualService is reported as a degraded `TrafficRouting` condition and no weights are applied.

## Cohort tracking

The facade tags each session with the rollout variant and cohort of the request that opened it. Three sources feed the tags:

- The `x-omnia-variant` request header, with the value `stable` or `candidate`.
- The `OMNIA_VARIANT` environment variable, which is `stable` on stable pods and `candidate` on candidate pods.
- The `x-omnia-cohort-id` request header, which carries any caller-supplied cohort identifier.

In `mesh` mode the owned VirtualService sets `x-omnia-variant` on each weighted destination, so the value follows the route taken. The facade prefers the header and falls back to `OMNIA_VARIANT`. In `replicaWeighted` mode nothing sets the header, so the pod's environment variable decides. The variant value is `candidate` even though the subset is named `canary`.

The controller never sets `x-omnia-cohort-id`. The facade reads it from the request and has no fallback, so it is empty unless something upstream sets it. Both values are stored on the session (`variant` and `cohort_id`) and recorded as the `omnia.variant` and `omnia.cohort.id` attributes on facade message spans. Because the facade accepts both headers from any caller, they identify a cohort for analysis and are not an access control.

## Sticky sessions

For experiments that require consistent user routing, configure `stickySession`:

```yaml
rollout:
  stickySession:
    hashOn: "x-user-id"
```

The controller sets a consistent hash on the header you name in the DestinationRule that you reference under `trafficRouting.istio`, so the same user always reaches the same variant. It removes the hash when the rollout ends. Sticky sessions need that `istio` reference. The DestinationRule the controller owns in `mesh` mode never gets a hash, and `replicaWeighted` has no routing layer, so neither mode applies `stickySession`. Each request is then routed independently by weight, and a user might see both variants across consecutive messages.

## Promotion and rollback

A rollout ends in one of three states.

Idle: when `rollout.candidate` matches the current spec (or is absent), no rollout is active. The controller maintains a single Deployment.

Promotion: when the rollout completes all steps, the candidate's config is merged into the main spec and the stable Deployment rolls to it **in the background**. Throughout that roll, 100% of traffic stays on the already-validated, warm candidate. Only once the stable Deployment is fully rolled out and healthy on the new config does traffic cut back to it and the candidate Deployment get deleted. No request is ever served from a cold or restarting pod, so promotion is zero-downtime. While the background roll is underway, `status.rollout.promoting` is `true` and the `RolloutActive` condition reports reason `Promoting`.

Rollback: the candidate is reverted to match the current spec, and the candidate Deployment is deleted. Traffic returns to 100% stable.

Rollback behavior is controlled by `rollout.rollback.mode`:

- `automatic` rolls back when an analysis step fails or the candidate Deployment exceeds its progress deadline.
- `manual` (the default) never rolls back on its own. A failed analysis leaves the rollout on its step, and you roll back by editing `rollout.candidate` to match the current spec.
- `disabled` behaves like `manual`.

With `manual`, a failed analysis leaves the rollout on its current step with `RolloutActive=True`, reason `AnalysisFailed`, and a warning event. The controller sets no timed requeue, but the rollout is not idle: the status write that records the failure triggers another reconcile, which re-runs the analysis from a fresh measurement sequence, so a metric that has recovered can pass and the rollout continues. The re-runs stop only when one fails on its first pass with the same message, because the status write is then unchanged and fires no event. The rollout then waits for the next event, such as an edit to the AgentRuntime. Nothing in the controller reads `disabled` separately, so a failed analysis leaves the rollout on its step exactly as it does under `manual`. The pod-health rollback runs only under `automatic`.

## Analysis integration

:::note[Enterprise]
RolloutAnalysis is an enterprise feature.
:::

Analysis steps run a `RolloutAnalysis` CRD, a reusable template whose metrics come from the `prometheus` provider, the `web` provider (an HTTP endpoint) or the `arenaEval` provider (an evaluation score). A Prometheus metric looks like this:

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: RolloutAnalysis
metadata:
  name: quality-check
spec:
  metrics:
    - name: error-rate
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(omnia_agent_requests_total{status="error", variant="candidate"}[5m]))
            /
            sum(rate(omnia_agent_requests_total{variant="candidate"}[5m]))
      successCondition: "result[0] < 0.05"
      count: 5
      failureLimit: 3
      interval: "1m"
```

The controller takes `count` measurements of each metric, spaced `interval` apart, and evaluates `successCondition` against each one:

- The analysis step **passes** (and the rollout advances) only once every metric has completed all `count` measurements without exceeding its `failureLimit`.
- The analysis step **fails** as soon as any metric's failed measurements exceed its `failureLimit`, which triggers the configured rollback behavior. With the default `failureLimit: 0`, the first failed measurement is decisive.
- A measurement that cannot be taken at all (Prometheus unreachable, template missing, malformed condition) is an **error**, not a failure. The controller retries every 30s and never rolls back on it, because the fault is in the observation path rather than in the candidate.

Measurement progress is recorded in the AgentRuntime's `status.rollout.analysis` and survives controller restarts. It is keyed on the step, the template, the template's spec revision and the candidate version. A change to any of them discards the progress and starts a fresh sequence, so a measurement taken against one candidate never counts against another.

An `arenaEval` metric reads the average score of one evaluation definition for the candidate's sessions. Until the evaluation has produced a result for the candidate, the measurement is an error and retries every 30s, so a young candidate is never rolled back for having no score yet.

## Related resources

- [Progressive Rollouts Tutorial](/tutorials/progressive-rollouts/): step-by-step walkthrough using `replicaWeighted` routing
- [AgentRuntime CRD Reference](/reference/core/agentruntime/#rollout): field-by-field specification
- [RolloutAnalysis CRD Reference](/reference/evaluation/rolloutanalysis/): analysis template fields