Set up observability

Enable the bundled observability stack and query your agents’ metrics, logs and traces in Grafana.

Prerequisites

Omnia installed with Helm, as in Install with Helm
kubectl access to the cluster

Enable the stack

Enable the components in your Helm values. All five default to false:
```
prometheus:
  enabled: true
grafana:
  enabled: true
loki:
  enabled: true
tempo:
  enabled: true
alloy:
  enabled: true
tracing:
  enabled: true
```
Each component toggles independently. tracing.enabled makes the operator pass the tracing settings to every agent. The endpoint resolves to <release>-alloy:4317 when Alloy is enabled, else <release>-tempo:4317. Set tracing.endpoint to use another collector.

Upgrade the release:

helm upgrade --install omnia oci://ghcr.io/altairalabs/charts/omnia \
  --devel \
  --namespace omnia-system \
  -f values.yaml

With grafana.enabled and observability.enabled (default true), the chart provisions Grafana datasources named Prometheus, Loki and Tempo for each enabled component.

Open Grafana

Port-forward the Grafana service. The service name is <release>-grafana:

kubectl port-forward svc/omnia-grafana 3000:80 -n omnia-system

Open http://localhost:3000 and sign in as admin. The password is grafana.adminPassword, which defaults to admin.

To reach Grafana through a gateway instead, set internalGateway.enabled: true (the gateway class defaults to istio). Then read the gateway address:

kubectl get gateway omnia-internal -n omnia-system -o jsonpath='{.status.addresses[0].value}'

Grafana is at http://<gateway-address>:8080/grafana/.

Query agent metrics

Open Explore, select the Prometheus datasource and run PromQL. Every agent series carries agent and namespace labels. See Platform metrics for every metric and label.

# Active WebSocket connections
omnia_agent_connections_active

# Request rate by outcome
sum by (status) (rate(omnia_agent_requests_total[5m]))

# P95 request latency
histogram_quantile(0.95, sum by (le) (rate(omnia_agent_request_duration_seconds_bucket[5m])))

# LLM cost per model, last hour
sum by (model) (increase(omnia_provider_cost_total[1h]))

# Tool call error ratio
sum(rate(omnia_tool_calls_total{status="error"}[5m])) / sum(rate(omnia_tool_calls_total[5m]))

# Median pipeline duration
histogram_quantile(0.5, sum by (le) (rate(omnia_pipeline_duration_seconds_bucket[5m])))

# Session API P99 latency
histogram_quantile(0.99, sum by (le) (rate(omnia_session_api_request_duration_seconds_bucket[5m])))

# Eval worker consumer lag per stream
omnia_eval_worker_stream_lag

Query agent logs

Alloy ships pod logs to Loki. Select the Loki datasource. Agent log lines carry the labels namespace, pod, container (facade or runtime) and agent (the AgentRuntime name):

{namespace="<agent-namespace>", container="runtime"}

{agent="my-agent"} |= "error"

Query agent traces

Tracing needs tracing.enabled: true from the first step. Select the Tempo datasource and search by service name (omnia-runtime-<agent>), trace ID, duration or tag. session.id is a span attribute on the conversation span.

Find slow conversations:

{ duration > 5s && resource.service.name =~ "omnia-runtime.*" }

Find failed tool calls:

{ name = "omnia.tool.call" && status = error }

To lower the sampling rate, set OMNIA_TRACING_SAMPLE_RATE (0.0 to 1.0, default 1.0) on the runtime container:

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: AgentRuntime
metadata:
  name: my-agent
spec:
  runtime:
    extraEnv:
      - name: OMNIA_TRACING_SAMPLE_RATE
        value: "0.1"

Span names and attributes are listed in Platform metrics.

Harden for production

Persist Prometheus data. Loki and Tempo persistence is on by default, so only the sizes change:

prometheus:
  server:
    persistentVolume:
      enabled: true
      size: 50Gi
loki:
  singleBinary:
    persistence:
      size: 50Gi
tempo:
  persistence:
    size: 10Gi

Replace the default Grafana password with a value, or with an existing Secret:

grafana:
  adminPassword: <your-password>

grafana:
  admin:
    existingSecret: grafana-admin-secret
    userKey: admin-user
    passwordKey: admin-password

Set resources on the components that need them (illustrative values):

prometheus:
  server:
    resources:
      requests: { cpu: 500m, memory: 512Mi }
      limits: { cpu: 1000m, memory: 1Gi }
grafana:
  resources:
    requests: { cpu: 100m, memory: 128Mi }
    limits: { cpu: 500m, memory: 256Mi }

The Loki ruler is off (loki.ruler.enabled: false). To alert on logs, enable it and configure ruler storage under loki.loki.rulerConfig, as the Loki chart documents.

Use an existing observability stack

Set enabled: false for the bundled prometheus, grafana, loki and tempo components. Agent pods carry no prometheus.io/* annotations. Each facade and runtime container declares a port named metrics, which your Prometheus must select. With the Prometheus Operator, set podMonitor.enabled: true and add the labels your Prometheus selects on:

podMonitor:
  enabled: true
  labels:
    release: kube-prometheus-stack

The session-api and eval-worker pods carry prometheus.io/scrape, prometheus.io/port and prometheus.io/path annotations.

Verify

In Grafana Explore with the Prometheus datasource, run omnia_agent_connections_active. A running agent returns one series per facade pod.
Run omnia_runtime_info. A running agent returns a series with value 1.
In Loki, run {agent="<agentruntime-name>"}. The agent’s log lines appear.
Send a message to the agent, then search Tempo for resource.service.name = "omnia-runtime-<agentruntime-name>". A omnia.runtime.conversation.turn span appears.

Troubleshooting

Agent series are missing from Prometheus

Open the Prometheus UI and check that the omnia-agents job lists the agent pod’s metrics ports as targets. The job keeps pods labelled omnia.altairalabs.ai/component=agent. With an external Prometheus, confirm the PodMonitor labels match your Prometheus selector.

No traces in Tempo

Check that the agent’s facade and runtime containers have OMNIA_TRACING_ENABLED=true and an OMNIA_TRACING_ENDPOINT:

kubectl get deployment <agentruntime-name> -n <agent-namespace> \
  -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="OMNIA_TRACING_ENDPOINT")].value}'

An empty result means tracing.enabled is false or no endpoint resolved.