# Set up observability

Enable the bundled observability stack and query your agents' metrics, logs and traces in Grafana.

## Prerequisites

- Omnia installed with Helm, as in [Install with Helm](/how-to/operations/install-helm/)
- `kubectl` access to the cluster

## Enable the stack

1. Enable the components in your Helm values. All five default to `false`:

   ```yaml
   prometheus:
     enabled: true
   grafana:
     enabled: true
   loki:
     enabled: true
   tempo:
     enabled: true
   alloy:
     enabled: true
   tracing:
     enabled: true
   ```

   Each component toggles independently. `tracing.enabled` makes the operator pass the tracing settings to every agent. The endpoint resolves to `<release>-alloy:4317` when Alloy is enabled, else `<release>-tempo:4317`. Set `tracing.endpoint` to use another collector.

2. Upgrade the release:

   ```bash
   helm upgrade --install omnia oci://ghcr.io/altairalabs/charts/omnia \
     --devel \
     --namespace omnia-system \
     -f values.yaml
   ```

With `grafana.enabled` and `observability.enabled` (default `true`), the chart provisions Grafana datasources named **Prometheus**, **Loki** and **Tempo** for each enabled component.

## Open Grafana

Port-forward the Grafana service. The service name is `<release>-grafana`:

```bash
kubectl port-forward svc/omnia-grafana 3000:80 -n omnia-system
```

Open `http://localhost:3000` and sign in as `admin`. The password is `grafana.adminPassword`, which defaults to `admin`.

To reach Grafana through a gateway instead, set `internalGateway.enabled: true` (the gateway class defaults to `istio`). Then read the gateway address:

```bash
kubectl get gateway omnia-internal -n omnia-system -o jsonpath='{.status.addresses[0].value}'
```

Grafana is at `http://<gateway-address>:8080/grafana/`.

## Query agent metrics

Open **Explore**, select the **Prometheus** datasource and run PromQL. Every agent series carries `agent` and `namespace` labels. See [Platform metrics](/reference/platform/observability-metrics/) for every metric and label.

```promql
# Active WebSocket connections
omnia_agent_connections_active

# Request rate by outcome
sum by (status) (rate(omnia_agent_requests_total[5m]))

# P95 request latency
histogram_quantile(0.95, sum by (le) (rate(omnia_agent_request_duration_seconds_bucket[5m])))

# LLM cost per model, last hour
sum by (model) (increase(omnia_provider_cost_total[1h]))

# Tool call error ratio
sum(rate(omnia_tool_calls_total{status="error"}[5m])) / sum(rate(omnia_tool_calls_total[5m]))

# Median pipeline duration
histogram_quantile(0.5, sum by (le) (rate(omnia_pipeline_duration_seconds_bucket[5m])))

# Session API P99 latency
histogram_quantile(0.99, sum by (le) (rate(omnia_session_api_request_duration_seconds_bucket[5m])))

# Eval worker consumer lag per stream
omnia_eval_worker_stream_lag
```

## Query agent logs

Alloy ships pod logs to Loki. Select the **Loki** datasource. Agent log lines carry the labels `namespace`, `pod`, `container` (`facade` or `runtime`) and `agent` (the AgentRuntime name):

```text
{namespace="<agent-namespace>", container="runtime"}

{agent="my-agent"} |= "error"
```

## Query agent traces

Tracing needs `tracing.enabled: true` from the first step. Select the **Tempo** datasource and search by service name (`omnia-runtime-<agent>`), trace ID, duration or tag. `session.id` is a span attribute on the conversation span.

Find slow conversations:

```text
{ duration > 5s && resource.service.name =~ "omnia-runtime.*" }
```

Find failed tool calls:

```text
{ name = "omnia.tool.call" && status = error }
```

To lower the sampling rate, set `OMNIA_TRACING_SAMPLE_RATE` (`0.0` to `1.0`, default `1.0`) on the runtime container:

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: AgentRuntime
metadata:
  name: my-agent
spec:
  runtime:
    extraEnv:
      - name: OMNIA_TRACING_SAMPLE_RATE
        value: "0.1"
```

Span names and attributes are listed in [Platform metrics](/reference/platform/observability-metrics/#traces).

## Harden for production

Persist Prometheus data. Loki and Tempo persistence is on by default, so only the sizes change:

```yaml
prometheus:
  server:
    persistentVolume:
      enabled: true
      size: 50Gi
loki:
  singleBinary:
    persistence:
      size: 50Gi
tempo:
  persistence:
    size: 10Gi
```

Replace the default Grafana password with a value, or with an existing Secret:

```yaml
grafana:
  adminPassword: <your-password>
```

```yaml
grafana:
  admin:
    existingSecret: grafana-admin-secret
    userKey: admin-user
    passwordKey: admin-password
```

Set resources on the components that need them (illustrative values):

```yaml
prometheus:
  server:
    resources:
      requests: { cpu: 500m, memory: 512Mi }
      limits: { cpu: 1000m, memory: 1Gi }
grafana:
  resources:
    requests: { cpu: 100m, memory: 128Mi }
    limits: { cpu: 500m, memory: 256Mi }
```

The Loki ruler is off (`loki.ruler.enabled: false`). To alert on logs, enable it and configure ruler storage under `loki.loki.rulerConfig`, as the Loki chart documents.

## Use an existing observability stack

Set `enabled: false` for the bundled `prometheus`, `grafana`, `loki` and `tempo` components. Agent pods carry no `prometheus.io/*` annotations. Each facade and runtime container declares a port named `metrics`, which your Prometheus must select. With the Prometheus Operator, set `podMonitor.enabled: true` and add the labels your Prometheus selects on:

```yaml
podMonitor:
  enabled: true
  labels:
    release: kube-prometheus-stack
```

The session-api and eval-worker pods carry `prometheus.io/scrape`, `prometheus.io/port` and `prometheus.io/path` annotations.

## Verify

1. In Grafana **Explore** with the **Prometheus** datasource, run `omnia_agent_connections_active`. A running agent returns one series per facade pod.
2. Run `omnia_runtime_info`. A running agent returns a series with value `1`.
3. In Loki, run `{agent="<agentruntime-name>"}`. The agent's log lines appear.
4. Send a message to the agent, then search Tempo for `resource.service.name = "omnia-runtime-<agentruntime-name>"`. A `omnia.runtime.conversation.turn` span appears.

## Troubleshooting

### Agent series are missing from Prometheus

Open the Prometheus UI and check that the `omnia-agents` job lists the agent pod's `metrics` ports as targets. The job keeps pods labelled `omnia.altairalabs.ai/component=agent`. With an external Prometheus, confirm the PodMonitor labels match your Prometheus selector.

### No traces in Tempo

Check that the agent's facade and runtime containers have `OMNIA_TRACING_ENABLED=true` and an `OMNIA_TRACING_ENDPOINT`:

```bash
kubectl get deployment <agentruntime-name> -n <agent-namespace> \
  -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="OMNIA_TRACING_ENDPOINT")].value}'
```

An empty result means `tracing.enabled` is false or no endpoint resolved.

## Related pages

- [Platform metrics](/reference/platform/observability-metrics/)
- [Memory metrics](/reference/platform/memory-metrics/)
- [Helm values](/reference/platform/helm-values/)
- [Policy engine](/explanation/security/policy-engine/)