Gate a rollout on managed Prometheus

Query Amazon Managed Service for Prometheus (AMP) or Azure Monitor managed Prometheus from a RolloutAnalysis metric, with no proxy in front of it.

The AgentRuntime controller runs the queries, so the cloud identity belongs to the operator pod. Field details are in the RolloutAnalysis reference.

Prerequisites

The Enterprise chart installed, so the RolloutAnalysis CRD exists.
An AMP workspace on EKS, or an Azure Monitor workspace on AKS, that receives the metrics you gate on.
Permission to create IAM roles (AWS) or managed identities and role assignments (Azure).

Amazon Managed Service for Prometheus

Create an IAM role for the operator’s ServiceAccount with IRSA. Its trust policy names the cluster’s OIDC provider and the subject system:serviceaccount:<release-namespace>:<operator-serviceaccount>. Its permissions policy allows aps:QueryMetrics on the workspace:
```
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["aps:QueryMetrics"],
    "Resource": "arn:aws:aps:eu-west-1:123456789012:workspace/ws-0123abcd"
  }]
}
```
EKS Pod Identity works as well: associate the role with the same ServiceAccount and skip the annotation in the next step.

Annotate the operator’s ServiceAccount with the role:

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/omnia-operator-amp

helm upgrade omnia omnia/omnia -n omnia-system -f values.yaml

The EKS webhook injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE into the operator pod when it restarts.

Set awsSigV4 on the metric. The address is the workspace endpoint without the /api/v1/query suffix. Its host must be aps-workspaces.<region>.amazonaws.com for the same region, or the API server rejects the object:

provider:
  prometheus:
    address: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-0123abcd
    awsSigV4:
      region: eu-west-1
    query: avg(omnia_eval_faithfulness{namespace="production"})

Azure Monitor managed Prometheus

Create a user-assigned managed identity and give it the Monitoring Data Reader role on the Azure Monitor workspace.
Add a federated credential to the identity. The issuer is the AKS cluster’s OIDC issuer URL, and the subject is system:serviceaccount:<release-namespace>:<operator-serviceaccount>.
Annotate the operator’s ServiceAccount with the identity’s client ID, and label the operator pod so the Workload Identity webhook mutates it:
values.yaml
```
serviceAccount:
  annotations:
    azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
operator:
  podOverrides:
    labels:
      azure.workload.identity/use: "true"
```
Terminal window
```
helm upgrade omnia omnia/omnia -n omnia-system -f values.yaml
```
The webhook injects AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_FEDERATED_TOKEN_FILE into the operator pod when it restarts.
Set azureMonitor on the metric. The address is the workspace’s query endpoint from its Overview page in the Azure portal. Its host must end in .prometheus.monitor.azure.com, and the token is always requested for the scope https://prometheus.monitor.azure.com/.default:
```
provider:
  prometheus:
    address: https://omnia-metrics-a1b2.eastus.prometheus.monitor.azure.com
    azureMonitor: {}
    query: avg(omnia_eval_faithfulness{namespace="production"})
```
To authenticate as an identity other than the one in AZURE_CLIENT_ID, set azureMonitor.clientID. That identity also needs a federated credential for the operator’s ServiceAccount.

Verify

Start a rollout whose steps include the analysis, then read its progress:

kubectl get agentruntime my-assistant -n production \
  -o jsonpath='{.status.rollout.message}{"\n"}'

A measurement that reached Prometheus shows up in status.rollout.analysis. An authentication failure is an error: the controller logs it, requeues after 30 seconds and does not roll back.

Troubleshooting

Symptom in the operator log	Cause
`load AWS credentials` or `retrieve AWS credentials`	The pod has no AWS identity. Check the ServiceAccount annotation and that the operator pod restarted after it was added.
`execute query: ... 403` from AMP	The role lacks `aps:QueryMetrics` on the workspace, or `region` differs from the workspace region.
`build Azure credential` or `acquire Azure token`	The Workload Identity webhook did not mutate the pod. Check the `azure.workload.identity/use` label and the federated credential subject.
`execute query: ... 403` from Azure Monitor	The identity lacks Monitoring Data Reader on the workspace.
`address must use https://` at apply time	The provider sets an auth block on an `http://` address.
`address host must be aps-workspaces.<awsSigV4.region>.amazonaws.com` at apply time	The address is not the AMP endpoint of the signing region, or is not in lower case.
`address host must end in .prometheus.monitor.azure.com` at apply time	The address is not an Azure Monitor workspace query endpoint, or is not in lower case.