# Realtime evals

Realtime evals run the same eval definitions that Arena uses before deploy against live production sessions. The definitions live in the PromptPack alongside validators and guardrails, and session events trigger execution automatically.

## Overview

A PromptPack author defines what good looks like once. The enterprise eval system applies that definition before deploy against synthetic scenarios (Arena batch jobs) and after deploy against live traffic (realtime evals). The assertion engine and the eval definitions are the same in both contexts, and only the input data differs.

Evals run asynchronously after the response is sent, so they add no latency to the conversation. Session events trigger them: the eval worker blocks on a Redis Streams read, and an inactivity check closes out idle sessions. Each AgentRuntime configures its own eval settings under `spec.evals`.

## Eval definitions

Evals are defined in the PromptPack's `pack.json`, either per prompt as a sibling to `validators` or at pack level. This example shows a prompt-level `evals` array:

```json
"evals": [
  {
    "id": "helpfulness",
    "type": "llm_judge",
    "trigger": "every_turn",
    "params": {
      "provider": "judge",
      "criteria": "Is the response helpful, accurate, and on-topic?",
      "rubric": "1-5 scale"
    }
  },
  {
    "id": "resolution",
    "type": "llm_judge",
    "trigger": "on_session_complete",
    "params": {
      "provider": "judge",
      "criteria": "Did the agent resolve the customer's issue?"
    }
  },
  {
    "id": "no-ssn-leak",
    "type": "regex",
    "trigger": "every_turn",
    "params": {
      "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b",
      "expect_match": false
    }
  }
]
```

The PromptPack reference lists the [eval fields, triggers and eval types](/reference/core/promptpack/#evals). Deterministic evals such as `regex` and `contains` cost nothing and can run on every session. LLM judge evals call a model, so each one costs money and takes seconds.

## Dual-pattern architecture

Realtime evals use two execution patterns, and the framework type of the agent decides which are available.

```mermaid
flowchart TB
    AR["AgentRuntime<br/><i>spec.evals.enabled: true</i>"]

    AR --> NonPK["Non-PromptKit<br/>(langchain, custom)"]
    AR --> PK["PromptKit agents"]

    NonPK --> PA1["<b>Pattern A only</b><br/>Facade → session-api<br/>→ Redis Streams → eval worker"]

    PK --> PA2["<b>Pattern A</b><br/>(same path as non-PromptKit)"]
    PK --> PC["<b>Pattern C</b><br/>runtime SDK eval middleware<br/>→ EventBus → OmniaEventStore"]
```

### Pattern A: platform events (all agents)

The following diagram shows how a recorded message becomes an eval result. The facade's RuntimeClient gRPC bus interceptor records conversation messages and sends them to session-api, which writes to PostgreSQL. For each assistant message, session-api publishes a lightweight event to the Redis stream `omnia:eval-events:<namespace>`. The eval worker consumes that stream, runs the evals, and writes results back through session-api.

```mermaid
flowchart LR
    F[Facade] -->|RuntimeClient interceptor| SA[session-api]
    SA --> PG[(PostgreSQL)]
    SA -.->|async publish| RS["Redis Streams<br/>omnia:eval-events:namespace"]
    RS --> EW[arena-eval-worker]
    EW -->|POST /api/v1/eval-results| SA
```

Pattern A is framework-agnostic because it records off the facade. It works with any runtime that implements the `omnia.runtime.v1` contract, including custom runtimes. This repo's tests verify it against PromptKit only. Community and custom runtimes are not covered.

### Pattern C: SDK eval middleware (PromptKit agents)

PromptKit agents add a second path inside the runtime. The runtime builds its PromptKit conversation with an eval runner and a group filter, and the SDK's eval middleware runs the matching evals after each turn. The runtime's `OmniaEventStore` receives the resulting eval events from the PromptKit EventBus and records them as `eval_results` rows, next to provider calls and pipeline events.

```mermaid
flowchart TB
    subgraph Runtime["Runtime container (PromptKit)"]
        RS[RecordingStage] --> EB[EventBus]
        EM[SDK eval middleware] --> Runner[EvalRunner]
        Runner -->|eval events| EB
    end

    EB --> OES[OmniaEventStore]
    OES --> SA[session-api]
```

Pattern C sees more than the facade does: per-call provider metadata, validation events and pipeline stage events, as well as tool calls, token counts and cost. Pattern A sees the recorded messages, tool calls, token counts, cost and total latency.

PromptKit agents run both patterns at once, and eval **groups** decide which path runs which eval. Inline results land in `eval_results` with `source="runtime-inline"` and worker results with `source="worker"`. The [session-api reference](/reference/platform/session-api/#eval-results) covers the table and its endpoints.

### Group-based routing

Every eval belongs to one or more groups. PromptKit classifies handlers into `fast-running` (deterministic, no network, such as `contains` and `regex`), `long-running` (LLM calls and slow I/O) and `external` (arbitrary external APIs). Every eval also carries the `default` group, and pack authors can add custom groups on the eval definition.

Each path runs only the evals whose groups match its filter:

```yaml
# AgentRuntime spec
evals:
  enabled: true
  inline:
    groups: ["fast-running"]                 # runs in the runtime process (Pattern C)
  worker:
    groups: ["long-running", "external"]     # runs out-of-band in the eval worker (Pattern A)
```

Those values are the defaults. Omnia substitutes them when a list is absent or empty, so an empty list is not an off-switch. Set `spec.evals.enabled: false` to disable evals. See the [AgentRuntime reference](/reference/core/agentruntime/#evals) for every field.

The defaults are disjoint, so an agent never runs the same eval twice. `default` appears in neither list because every eval carries it: adding it to the inline filter would route every long-running handler, such as `llm_judge`, onto the turn path. PromptKit classifies any handler that is neither long-running nor external as `fast-running`, so user-defined lightweight handlers need no extra registration. If you override the defaults so that they overlap, you get two `eval_results` rows for the same turn, one per `source`. That is occasionally useful for comparing inline against reloaded-session behaviour.

## Eval worker

The eval worker is a long-running Deployment that consumes the Redis stream and runs the evals for Pattern A. The operator provisions one per service group, named `arena-eval-worker-<group>`, rather than one cluster-wide worker. A multi-workspace cluster therefore runs several workers, each scoped to its group's agents. Each worker reads the stream for the namespace it is deployed into. The [Helm values reference](/reference/platform/helm-values/#eval-worker-configuration) covers the worker image.

### When a worker is created

The operator creates a group's worker when the group has a **non-PromptKit**, eval-enabled AgentRuntime. PromptKit agents run their `fast-running` evals inline through Pattern C, so a group made up entirely of PromptKit agents gets no worker, and its `long-running` and `external` evals (for example `llm_judge`) would run nowhere.

To force a worker for such a group, opt in on the service group:

```yaml
# Workspace spec, a service group
evalWorker:
  enabled: true          # force an eval worker even for all-PromptKit groups
```

This matters for rollout gating. Only the worker tags `omnia_eval_*` metrics with the `variant` label that `RolloutAnalysis` gates key on, so a PromptKit agent behind a variant-gated eval needs a worker to produce its gate metric. `enabled` has no effect when the group has no eval-enabled agent, because there is nothing to evaluate.

### Pod configuration and RBAC

The service group's `evalWorker.podOverrides` configures the worker pod, mirroring the memory and session `podOverrides` on the group. A typical use points the worker at a shared ServiceAccount (for example one labelled `azure.workload.identity/use: "true"`) so that `llm_judge` can mint a federated token for a keyless cloud judge provider. Without it the judge LLM is never called and every score is `0`. The operator binds the worker's Role and the workspace-reader ClusterRole to the effective ServiceAccount, so an overridden ServiceAccount keeps working. The Role grants `get` and `list` on `configmaps`, `secrets`, `agentruntimes`, `providers` and `promptpacks`.

### What the worker does

The worker joins a Redis consumer group, so several replicas share the stream. For each event it reads the agent's AgentRuntime for eval configuration, resolves the PromptPack to its ConfigMap and loads the eval definitions (cached per pack version), and fetches the session's messages from session-api. It then runs the evals with the PromptKit eval engine and writes the results to session-api.

### Metrics

The worker serves its `omnia_eval_*` metrics (scores, judge cost and others) at `/metrics` on port `9090`. The operator stamps each worker pod with the `app.kubernetes.io/component=eval-worker` label and `prometheus.io/{scrape,port,path}` annotations. A scrape job discovers workers by label across namespaces, so metrics follow new workers automatically. These metrics feed variant-gated `RolloutAnalysis` gates.

### Why a Deployment and not Jobs

Arena batch evaluation uses Jobs because each ArenaJob is a discrete unit of work. Realtime evals are continuous, and a Job per session event would add 5-15 seconds of pod scheduling to every eval. A persistent per-group worker processes events in milliseconds and scales horizontally through the consumer group.

## Judge provider resolution

LLM judge evals need a provider. The worker builds one from each entry in the AgentRuntime's `spec.providers` list and passes the set to the PromptKit eval engine, keyed by the entry's `name`:

```yaml
spec:
  providers:
    - name: default
      providerRef:
        name: claude-sonnet        # primary LLM for the agent
    - name: judge
      providerRef:
        name: claude-haiku         # cheap, fast model for eval judging
```

An eval chooses its judge with the `provider` param, as in the example above. Without that param, the engine uses the first entry in alphabetical order by name, which in this example is `default`, the agent's own model. Name the judge explicitly. Because the param selects a name, one agent can use a cheap judge for `every_turn` evals and a stronger one for `on_session_complete`.

The runtime does not resolve judge providers, so `llm_judge` evals run on the worker path.

## Session completion

Session-level evals need to know when a session has ended. When session-api moves a session's status to `completed`, it publishes a `session.completed` event. The worker also records the time of each assistant message per session and checks every 30 seconds, and a session with no assistant message for 5 minutes counts as complete.

The tracker lives in worker memory. A worker restart drops the idle timers for sessions it was tracking, and those sessions then complete only through an explicit close.

## Controlling cost

LLM judge evals cost money, and an uncontrolled judge on every turn adds up. Three levers limit the spend.

Groups keep expensive evals on the worker path, where they run out-of-band, and cheap ones inline. The `sample_turns` and `sample_sessions` triggers run an eval on a hash-sampled share of turns or sessions, set by `sample_percentage` on the eval definition. For judge choice, use a cheap, fast model such as Claude Haiku for `every_turn` evals and a stronger model such as Claude Sonnet for `on_session_complete` evals.

## Result storage and the dashboard

session-api stores results in the `eval_results` table. Each row records the session and message evaluated, the eval ID, type and trigger, the pass/fail verdict where one exists, a numeric score, the duration and the `source`. Judge token usage and cost are recorded as `provider_calls` rows. The [session-api reference](/reference/platform/session-api/#eval-results) lists the endpoints.

The dashboard has two views of the results. The session detail view shows the checks for each assistant message inline in the review. The Quality page (`/quality`) shows an eval count, the average and lowest score, a score trend and a per-eval breakdown, filtered by time range, agent and PromptPack.

The dashboard reads them through management-api, whose routes under `/api/v1/workspaces/{name}/eval-results` call session-api.

## Design decisions

### Why per-agent eval config?

Eval configuration lives on the AgentRuntime, not in a global setting or a separate CRD, for these reasons:

- Evals are tied to the agent's PromptPack, which is already on the AgentRuntime.
- Judge providers differ per agent: cheap judges for low-stakes agents, strong judges for critical ones.
- Operators enable and disable evals per agent, not globally.

### Why two patterns?

Pattern A gives universal coverage for any framework. Pattern C gives PromptKit agents richer event data. Supporting both means evals work whatever the framework, while PromptKit users get the enhanced data.

## Related pages

- [Configure realtime evals](/how-to/evaluation/configure-realtime-evals/) for the setup steps.
- [AgentRuntime reference](/reference/core/agentruntime/#evals) for every `spec.evals` field.
- [PromptPack reference](/reference/core/promptpack/#evals) for eval definition fields, triggers and types.
- [RolloutAnalysis reference](/reference/evaluation/rolloutanalysis/) for gating a rollout on `omnia_eval_*` metrics.