Platform metrics

Metric names include the Prometheus _total and _seconds suffixes. For the memory services, see Memory metrics.

Scrape targets

Component	Port	Discovery
Facade container	8081, named metrics	Pod container port named metrics
Runtime container	9001, named metrics	Pod container port named metrics
Policy broker sidecar (Enterprise)	8091, named metrics	Pod container port named metrics
Eval worker	9090	Pod annotations prometheus.io/scrape, prometheus.io/port, prometheus.io/path
Arena worker (Enterprise)	9090	Pod annotations prometheus.io/scrape, prometheus.io/port, prometheus.io/path
Session API	9090	Pod annotations prometheus.io/scrape, prometheus.io/port, prometheus.io/path

Agent pods carry no prometheus.io/* annotations. The bundled Prometheus discovers them with the omnia-agents job, which keeps pods labelled omnia.altairalabs.ai/component=agent and their container ports named metrics. An external Prometheus Operator discovers them with the PodMonitor that podMonitor.enabled: true renders.

Every series from the facade, runtime and policy broker carries the agent and namespace labels. The runtime’s PromptKit SDK series also carry promptpack_name; omnia_runtime_info does not.

Facade

Metric	Type	Labels	Meaning
omnia_agent_connections_active	Gauge	none	Active WebSocket connections
omnia_agent_connections_total	Counter	none	WebSocket connections since startup
omnia_agent_sessions_active	Gauge	none	Active sessions
omnia_agent_session_store	Gauge	mode	1 for the session store in use (httpclient or none). none means no session-api archive is configured, so the facade records no session, token or cost data.
omnia_agent_requests_inflight	Gauge	none	Requests being processed (pending LLM calls)
omnia_agent_requests_total	Counter	status, variant	Requests processed. status is success or error.
omnia_agent_request_duration_seconds	Histogram	handler, variant	Request processing time
omnia_agent_messages_received_total	Counter	none	WebSocket messages received
omnia_agent_messages_sent_total	Counter	none	WebSocket messages sent

variant is stable or candidate and distinguishes the two sides of a rollout.

Runtime

The runtime registers the PromptKit SDK collector under the omnia prefix.

Metric	Type	Labels	Meaning
omnia_pipeline_duration_seconds	Histogram	status	Pipeline execution time
omnia_provider_requests_total	Counter	provider, model, source, status	LLM provider calls
omnia_provider_request_duration_seconds	Histogram	provider, model, source	LLM provider call time
omnia_provider_input_tokens_total	Counter	provider, model, source	Input tokens sent
omnia_provider_output_tokens_total	Counter	provider, model, source	Output tokens received
omnia_provider_cached_tokens_total	Counter	provider, model, source	Cached tokens
omnia_provider_cost_total	Counter	provider, model, source	Cost in USD
omnia_tool_calls_total	Counter	tool, status	Tool calls. status is success or error; a call whose backend failed, timed out or was refused by the circuit breaker is an error.
omnia_tool_call_duration_seconds	Histogram	tool	Tool call time
omnia_validations_total	Counter	validator, validator_type, status	Validation checks. status is passed or failed.
omnia_validation_duration_seconds	Histogram	validator, validator_type	Validation time
omnia_runtime_info	Gauge	none	Always 1; confirms the runtime is scraped

The collector also emits omnia_tts_*, omnia_stt_*, omnia_embedding_*, omnia_image_gen_* and omnia_inference_* families with the same provider, model and source labels.

Eval worker

The operator deploys one eval worker per service group when a non-PromptKit, eval-enabled AgentRuntime exists in the workspace. It also deploys one for a PromptKit agent whose service group sets evalWorker.enabled.

Metric	Type	Labels	Meaning
omnia_eval_worker_events_received_total	Counter	event_type	Session events consumed from Redis Streams
omnia_eval_worker_event_processing_duration_seconds	Histogram	event_type	Time to process one stream event
omnia_eval_worker_stream_lag	Gauge	stream	Pending messages per Redis stream
omnia_eval_worker_evals_executed_total	Counter	eval_type, trigger, status	Eval executions. status is success or error.
omnia_eval_worker_eval_duration_seconds	Histogram	eval_type, trigger	Eval execution time
omnia_eval_worker_evals_sampled_total	Counter	eval_type, decision	Sampling decisions: sampled or skipped
omnia_eval_worker_evals_skipped_suppressed_total	Counter	trigger	Evals skipped because the session content was suppressed for an opted-out subject
omnia_eval_worker_results_written_total	Counter	status	Eval results written to session-api

Arena worker

Enterprise. Each ArenaRun’s worker pods serve these on /metrics. The job_name label holds the ArenaRun name.

Metric	Type	Labels	Meaning
omnia_arena_work_items_total	Counter	job_name, status	Work items processed. status is pass or fail.
omnia_arena_work_item_duration_seconds	Histogram	job_name	Work item duration (0.1 s to 600 s buckets)
omnia_arena_turn_latency_seconds	Histogram	job_name, scenario, provider	The work item’s duration, per scenario and provider
omnia_arena_ttft_seconds	Histogram	job_name, scenario, provider	Time to first token, when the item reports one
omnia_arena_trials_total	Counter	job_name, scenario, provider, status	Trials by outcome. status is pass or fail.
omnia_arena_errors_total	Counter	job_name, provider, error_type	Errors by type
omnia_arena_tokens_total	Counter	job_name, provider, direction	Tokens. direction is input or output.
omnia_arena_active_vus	Gauge	none	Active virtual users
omnia_arena_queue_items	Gauge	status	Queue items by status: pending, processing, completed or failed
omnia_arena_queue_operations_total	Counter	operation, status	Queue operations. operation is push, pop, ack, nack, complete_item or fail_item. status is success or error.
omnia_arena_queue_operation_duration_seconds	Histogram	operation	Queue operation time
omnia_arena_queue_jobs_active	Gauge	none	Active jobs
omnia_arena_queue_retries_total	Counter	none	Item retry attempts

For queries and an alert on failing work items, see Monitor Arena jobs.

Session API

Metric	Type	Labels	Meaning
omnia_session_api_requests_total	Counter	method, route, status_code	HTTP requests
omnia_session_api_request_duration_seconds	Histogram	method, route, status_code	HTTP request time
omnia_session_api_events_published_total	Counter	status	Redis stream publish attempts. status is success or error.
omnia_session_api_event_publish_duration_seconds	Histogram	none	Time to publish one event to Redis Streams

Policy broker

Enterprise. The broker sidecar serves these on /metrics.

Metric	Type	Labels	Meaning
omnia_toolpolicy_decisions_total	Counter	outcome, tool_registry, policy	ToolPolicy decisions. outcome is allowed, denied, would_deny or error. policy is the ToolPolicy that produced the decision: the one that granted an allow or refused a deny. It is empty when no ToolPolicy matched the call, when no policy granted it (no-matching-permit), and when only audit-mode policies matched and none would deny.
omnia_toolpolicy_decision_duration_seconds	Histogram	none	Broker decision time (0.5 ms to 0.5 s buckets)
omnia_toolpolicy_active_policies	Gauge	none	ToolPolicies compiled and loaded by the broker

The metrics carry no per-rule detail. The rule that fired is in the broker’s policy_decision log line. See Policy engine.

Traces

Runtime spans use the tracer omnia-runtime and the service name omnia-runtime-<agent>.

Span	Attributes
omnia.runtime.conversation.turn	session.id, omnia.turn.index, omnia.promptpack.name, omnia.promptpack.version, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cost, gen_ai.prompt.length, gen_ai.response.length
omnia.runtime.function.invoke (function mode)	omnia.invocation.id, omnia.mode, omnia.promptpack.name, omnia.promptpack.version
genai.chat	gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cost
omnia.tool.call	tool.name, tool.registry.name, tool.registry.namespace, tool.handler.name, tool.handler.type, tool.duration_ms, tool.request.bytes, tool.response.bytes

A failed tool call sets the span status to error and records an exception event. The facade emits omnia.facade.message and omnia.facade.a2a.message; SERVICES.md in the repository root lists the full inventory.

Tracing environment variables

Variable	Meaning	Default
OMNIA_TRACING_ENABLED	Enables OpenTelemetry export	false
OMNIA_TRACING_ENDPOINT	OTLP gRPC endpoint (host:port)	empty
OMNIA_TRACING_SAMPLE_RATE	Root sampling rate, 0.0 to 1.0. 0 samples no root spans.	1.0
OMNIA_TRACING_INSECURE	Disables TLS for the OTLP connection	false

The operator sets OMNIA_TRACING_ENABLED, OMNIA_TRACING_ENDPOINT and OMNIA_TRACING_INSECURE on the facade and runtime containers when the chart value tracing.enabled is true and an endpoint resolves.