Realtime evals

Realtime evals run the same eval definitions that Arena uses before deploy against live production sessions. The definitions live in the PromptPack alongside validators and guardrails, and session events trigger execution automatically.

Overview

A PromptPack author defines what good looks like once. The enterprise eval system applies that definition before deploy against synthetic scenarios (Arena batch jobs) and after deploy against live traffic (realtime evals). The assertion engine and the eval definitions are the same in both contexts, and only the input data differs.

Evals run asynchronously after the response is sent, so they add no latency to the conversation. Session events trigger them: the eval worker blocks on a Redis Streams read, and an inactivity check closes out idle sessions. Each AgentRuntime configures its own eval settings under spec.evals.

Eval definitions

Evals are defined in the PromptPack’s pack.json, either per prompt as a sibling to validators or at pack level. This example shows a prompt-level evals array:

"evals": [
  {
    "id": "helpfulness",
    "type": "llm_judge",
    "trigger": "every_turn",
    "params": {
      "provider": "judge",
      "criteria": "Is the response helpful, accurate, and on-topic?",
      "rubric": "1-5 scale"
    }
  },
  {
    "id": "resolution",
    "type": "llm_judge",
    "trigger": "on_session_complete",
    "params": {
      "provider": "judge",
      "criteria": "Did the agent resolve the customer's issue?"
    }
  },
  {
    "id": "no-ssn-leak",
    "type": "regex",
    "trigger": "every_turn",
    "params": {
      "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b",
      "expect_match": false
    }
  }
]

The PromptPack reference lists the eval fields, triggers and eval types. Deterministic evals such as regex and contains cost nothing and can run on every session. LLM judge evals call a model, so each one costs money and takes seconds.

Dual-pattern architecture

Realtime evals use two execution patterns, and the framework type of the agent decides which are available.

flowchart TB
    AR["AgentRuntime<br/><i>spec.evals.enabled: true</i>"]

    AR --> NonPK["Non-PromptKit<br/>(langchain, custom)"]
    AR --> PK["PromptKit agents"]

    NonPK --> PA1["<b>Pattern A only</b><br/>Facade → session-api<br/>→ Redis Streams → eval worker"]

    PK --> PA2["<b>Pattern A</b><br/>(same path as non-PromptKit)"]
    PK --> PC["<b>Pattern C</b><br/>runtime SDK eval middleware<br/>→ EventBus → OmniaEventStore"]

Pattern A: platform events (all agents)

The following diagram shows how a recorded message becomes an eval result. The facade’s RuntimeClient gRPC bus interceptor records conversation messages and sends them to session-api, which writes to PostgreSQL. For each assistant message, session-api publishes a lightweight event to the Redis stream omnia:eval-events:<namespace>. The eval worker consumes that stream, runs the evals, and writes results back through session-api.

flowchart LR
    F[Facade] -->|RuntimeClient interceptor| SA[session-api]
    SA --> PG[(PostgreSQL)]
    SA -.->|async publish| RS["Redis Streams<br/>omnia:eval-events:namespace"]
    RS --> EW[arena-eval-worker]
    EW -->|POST /api/v1/eval-results| SA

Pattern A is framework-agnostic because it records off the facade. It works with any runtime that implements the omnia.runtime.v1 contract, including custom runtimes. This repo’s tests verify it against PromptKit only. Community and custom runtimes are not covered.

Pattern C: SDK eval middleware (PromptKit agents)

PromptKit agents add a second path inside the runtime. The runtime builds its PromptKit conversation with an eval runner and a group filter, and the SDK’s eval middleware runs the matching evals after each turn. The runtime’s OmniaEventStore receives the resulting eval events from the PromptKit EventBus and records them as eval_results rows, next to provider calls and pipeline events.

flowchart TB
    subgraph Runtime["Runtime container (PromptKit)"]
        RS[RecordingStage] --> EB[EventBus]
        EM[SDK eval middleware] --> Runner[EvalRunner]
        Runner -->|eval events| EB
    end

    EB --> OES[OmniaEventStore]
    OES --> SA[session-api]

Pattern C sees more than the facade does: per-call provider metadata, validation events and pipeline stage events, as well as tool calls, token counts and cost. Pattern A sees the recorded messages, tool calls, token counts, cost and total latency.

PromptKit agents run both patterns at once, and eval groups decide which path runs which eval. Inline results land in eval_results with source="runtime-inline" and worker results with source="worker". The session-api reference covers the table and its endpoints.

Group-based routing

Every eval belongs to one or more groups. PromptKit classifies handlers into fast-running (deterministic, no network, such as contains and regex), long-running (LLM calls and slow I/O) and external (arbitrary external APIs). Every eval also carries the default group, and pack authors can add custom groups on the eval definition.

Each path runs only the evals whose groups match its filter:

# AgentRuntime spec
evals:
  enabled: true
  inline:
    groups: ["fast-running"]                 # runs in the runtime process (Pattern C)
  worker:
    groups: ["long-running", "external"]     # runs out-of-band in the eval worker (Pattern A)

Those values are the defaults. Omnia substitutes them when a list is absent or empty, so an empty list is not an off-switch. Set spec.evals.enabled: false to disable evals. See the AgentRuntime reference for every field.

The defaults are disjoint, so an agent never runs the same eval twice. default appears in neither list because every eval carries it: adding it to the inline filter would route every long-running handler, such as llm_judge, onto the turn path. PromptKit classifies any handler that is neither long-running nor external as fast-running, so user-defined lightweight handlers need no extra registration. If you override the defaults so that they overlap, you get two eval_results rows for the same turn, one per source. That is occasionally useful for comparing inline against reloaded-session behaviour.

Eval worker

The eval worker is a long-running Deployment that consumes the Redis stream and runs the evals for Pattern A. The operator provisions one per service group, named arena-eval-worker-<group>, rather than one cluster-wide worker. A multi-workspace cluster therefore runs several workers, each scoped to its group’s agents. Each worker reads the stream for the namespace it is deployed into. The Helm values reference covers the worker image.

When a worker is created

The operator creates a group’s worker when the group has a non-PromptKit, eval-enabled AgentRuntime. PromptKit agents run their fast-running evals inline through Pattern C, so a group made up entirely of PromptKit agents gets no worker, and its long-running and external evals (for example llm_judge) would run nowhere.

To force a worker for such a group, opt in on the service group:

# Workspace spec, a service group
evalWorker:
  enabled: true          # force an eval worker even for all-PromptKit groups

This matters for rollout gating. Only the worker tags omnia_eval_* metrics with the variant label that RolloutAnalysis gates key on, so a PromptKit agent behind a variant-gated eval needs a worker to produce its gate metric. enabled has no effect when the group has no eval-enabled agent, because there is nothing to evaluate.

Pod configuration and RBAC

The service group’s evalWorker.podOverrides configures the worker pod, mirroring the memory and session podOverrides on the group. A typical use points the worker at a shared ServiceAccount (for example one labelled azure.workload.identity/use: "true") so that llm_judge can mint a federated token for a keyless cloud judge provider. Without it the judge LLM is never called and every score is 0. The operator binds the worker’s Role and the workspace-reader ClusterRole to the effective ServiceAccount, so an overridden ServiceAccount keeps working. The Role grants get and list on configmaps, secrets, agentruntimes, providers and promptpacks.

What the worker does

The worker joins a Redis consumer group, so several replicas share the stream. For each event it reads the agent’s AgentRuntime for eval configuration, resolves the PromptPack to its ConfigMap and loads the eval definitions (cached per pack version), and fetches the session’s messages from session-api. It then runs the evals with the PromptKit eval engine and writes the results to session-api.

Metrics

The worker serves its omnia_eval_* metrics (scores, judge cost and others) at /metrics on port 9090. The operator stamps each worker pod with the app.kubernetes.io/component=eval-worker label and prometheus.io/{scrape,port,path} annotations. A scrape job discovers workers by label across namespaces, so metrics follow new workers automatically. These metrics feed variant-gated RolloutAnalysis gates.

Why a Deployment and not Jobs

Arena batch evaluation uses Jobs because each ArenaJob is a discrete unit of work. Realtime evals are continuous, and a Job per session event would add 5-15 seconds of pod scheduling to every eval. A persistent per-group worker processes events in milliseconds and scales horizontally through the consumer group.

Judge provider resolution

LLM judge evals need a provider. The worker builds one from each entry in the AgentRuntime’s spec.providers list and passes the set to the PromptKit eval engine, keyed by the entry’s name:

spec:
  providers:
    - name: default
      providerRef:
        name: claude-sonnet        # primary LLM for the agent
    - name: judge
      providerRef:
        name: claude-haiku         # cheap, fast model for eval judging

An eval chooses its judge with the provider param, as in the example above. Without that param, the engine uses the first entry in alphabetical order by name, which in this example is default, the agent’s own model. Name the judge explicitly. Because the param selects a name, one agent can use a cheap judge for every_turn evals and a stronger one for on_session_complete.

The runtime does not resolve judge providers, so llm_judge evals run on the worker path.

Session completion

Session-level evals need to know when a session has ended. When session-api moves a session’s status to completed, it publishes a session.completed event. The worker also records the time of each assistant message per session and checks every 30 seconds, and a session with no assistant message for 5 minutes counts as complete.

The tracker lives in worker memory. A worker restart drops the idle timers for sessions it was tracking, and those sessions then complete only through an explicit close.

Controlling cost

LLM judge evals cost money, and an uncontrolled judge on every turn adds up. Three levers limit the spend.

Groups keep expensive evals on the worker path, where they run out-of-band, and cheap ones inline. The sample_turns and sample_sessions triggers run an eval on a hash-sampled share of turns or sessions, set by sample_percentage on the eval definition. For judge choice, use a cheap, fast model such as Claude Haiku for every_turn evals and a stronger model such as Claude Sonnet for on_session_complete evals.

Result storage and the dashboard

session-api stores results in the eval_results table. Each row records the session and message evaluated, the eval ID, type and trigger, the pass/fail verdict where one exists, a numeric score, the duration and the source. Judge token usage and cost are recorded as provider_calls rows. The session-api reference lists the endpoints.

The dashboard has two views of the results. The session detail view shows the checks for each assistant message inline in the review. The Quality page (/quality) shows an eval count, the average and lowest score, a score trend and a per-eval breakdown, filtered by time range, agent and PromptPack.

The dashboard reads them through management-api, whose routes under /api/v1/workspaces/{name}/eval-results call session-api.

Design decisions

Why per-agent eval config?

Eval configuration lives on the AgentRuntime, not in a global setting or a separate CRD, for these reasons:

• Evals are tied to the agent’s PromptPack, which is already on the AgentRuntime.
• Judge providers differ per agent: cheap judges for low-stakes agents, strong judges for critical ones.
• Operators enable and disable evals per agent, not globally.

Why two patterns?

Pattern A gives universal coverage for any framework. Pattern C gives PromptKit agents richer event data. Supporting both means evals work whatever the framework, while PromptKit users get the enhanced data.

Related pages

• Configure realtime evals for the setup steps.
• AgentRuntime reference for every spec.evals field.
• PromptPack reference for eval definition fields, triggers and types.
• RolloutAnalysis reference for gating a rollout on omnia_eval_* metrics.