# Gate a rollout on managed Prometheus

Query Amazon Managed Service for Prometheus (AMP) or Azure Monitor managed
Prometheus from a `RolloutAnalysis` metric, with no proxy in front of it.

The AgentRuntime controller runs the queries, so the cloud identity belongs to
the operator pod. Field details are in the
[RolloutAnalysis reference](/reference/evaluation/rolloutanalysis/#providerprometheus).

## Prerequisites

- The Enterprise chart installed, so the `RolloutAnalysis` CRD exists.
- An AMP workspace on EKS, or an Azure Monitor workspace on AKS, that receives
  the metrics you gate on.
- Permission to create IAM roles (AWS) or managed identities and role
  assignments (Azure).

## Amazon Managed Service for Prometheus

1. Create an IAM role for the operator's ServiceAccount with IRSA. Its trust
   policy names the cluster's OIDC provider and the subject
   `system:serviceaccount:<release-namespace>:<operator-serviceaccount>`. Its
   permissions policy allows `aps:QueryMetrics` on the workspace:

   ```json
   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": ["aps:QueryMetrics"],
       "Resource": "arn:aws:aps:eu-west-1:123456789012:workspace/ws-0123abcd"
     }]
   }
   ```

   EKS Pod Identity works as well: associate the role with the same
   ServiceAccount and skip the annotation in the next step.

2. Annotate the operator's ServiceAccount with the role:

   ```yaml
   # values.yaml
   serviceAccount:
     annotations:
       eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/omnia-operator-amp
   ```

   ```bash
   helm upgrade omnia omnia/omnia -n omnia-system -f values.yaml
   ```

   The EKS webhook injects `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`
   into the operator pod when it restarts.

3. Set `awsSigV4` on the metric. The `address` is the workspace endpoint
   without the `/api/v1/query` suffix. Its host must be
   `aps-workspaces.<region>.amazonaws.com` for the same `region`, or the API
   server rejects the object:

   ```yaml
   provider:
     prometheus:
       address: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-0123abcd
       awsSigV4:
         region: eu-west-1
       query: avg(omnia_eval_faithfulness{namespace="production"})
   ```

## Azure Monitor managed Prometheus

1. Create a user-assigned managed identity and give it the **Monitoring Data
   Reader** role on the Azure Monitor workspace.

2. Add a federated credential to the identity. The issuer is the AKS cluster's
   OIDC issuer URL, and the subject is
   `system:serviceaccount:<release-namespace>:<operator-serviceaccount>`.

3. Annotate the operator's ServiceAccount with the identity's client ID, and
   label the operator pod so the Workload Identity webhook mutates it:

   ```yaml
   # values.yaml
   serviceAccount:
     annotations:
       azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
   operator:
     podOverrides:
       labels:
         azure.workload.identity/use: "true"
   ```

   ```bash
   helm upgrade omnia omnia/omnia -n omnia-system -f values.yaml
   ```

   The webhook injects `AZURE_CLIENT_ID`, `AZURE_TENANT_ID` and
   `AZURE_FEDERATED_TOKEN_FILE` into the operator pod when it restarts.

4. Set `azureMonitor` on the metric. The `address` is the workspace's query
   endpoint from its Overview page in the Azure portal. Its host must end in
   `.prometheus.monitor.azure.com`, and the token is always requested for the
   scope `https://prometheus.monitor.azure.com/.default`:

   ```yaml
   provider:
     prometheus:
       address: https://omnia-metrics-a1b2.eastus.prometheus.monitor.azure.com
       azureMonitor: {}
       query: avg(omnia_eval_faithfulness{namespace="production"})
   ```

   To authenticate as an identity other than the one in `AZURE_CLIENT_ID`,
   set `azureMonitor.clientID`. That identity also needs a federated
   credential for the operator's ServiceAccount.

## Verify

Start a rollout whose steps include the analysis, then read its progress:

```bash
kubectl get agentruntime my-assistant -n production \
  -o jsonpath='{.status.rollout.message}{"\n"}'
```

A measurement that reached Prometheus shows up in
`status.rollout.analysis`. An authentication failure is an error: the
controller logs it, requeues after 30 seconds and does not roll back.

## Troubleshooting

| Symptom in the operator log | Cause |
|---|---|
| `load AWS credentials` or `retrieve AWS credentials` | The pod has no AWS identity. Check the ServiceAccount annotation and that the operator pod restarted after it was added. |
| `execute query: ... 403` from AMP | The role lacks `aps:QueryMetrics` on the workspace, or `region` differs from the workspace region. |
| `build Azure credential` or `acquire Azure token` | The Workload Identity webhook did not mutate the pod. Check the `azure.workload.identity/use` label and the federated credential subject. |
| `execute query: ... 403` from Azure Monitor | The identity lacks Monitoring Data Reader on the workspace. |
| `address must use https://` at apply time | The provider sets an auth block on an `http://` address. |
| `address host must be aps-workspaces.<awsSigV4.region>.amazonaws.com` at apply time | The address is not the AMP endpoint of the signing region, or is not in lower case. |
| `address host must end in .prometheus.monitor.azure.com` at apply time | The address is not an Azure Monitor workspace query endpoint, or is not in lower case. |

## Related

- [RolloutAnalysis CRD](/reference/evaluation/rolloutanalysis/)
- [Roll back a rollout](/how-to/agents/roll-back-a-rollout/)
- [Configure pod overrides](/how-to/operations/configure-pod-overrides/)