# Configure realtime evals

Enable realtime evals on an AgentRuntime so that live conversations are scored against the eval definitions in its PromptPack.

## Prerequisites

- A session-api with PostgreSQL and Redis in the agent's service group. session-api publishes session events to a Redis stream and stores results in the `eval_results` table.
- A PromptPack that the agent already uses.
- A Provider for the judge model, if you use `llm_judge` evals. The Provider's role must be `llm`.

## Enable evals

1. Set `evals.enabled` on the AgentRuntime:

   ```yaml
   apiVersion: omnia.altairalabs.ai/v1alpha1
   kind: AgentRuntime
   metadata:
     name: my-agent
   spec:
     evals:
       enabled: true
   ```

   Without other settings, deterministic evals (`contains`, `regex`) run inside the runtime and LLM-judge and external evals run in the eval-worker. To change that split, set `evals.inline.groups` and `evals.worker.groups`. See the [AgentRuntime reference](/reference/core/agentruntime/#evals) for both fields and their defaults.

2. If the agent uses the PromptKit framework (the default) and the pack has `llm_judge` or other external evals, opt the service group into an eval-worker. The operator creates a worker for non-PromptKit agents without this step.

   ```yaml
   apiVersion: omnia.altairalabs.ai/v1alpha1
   kind: Workspace
   spec:
     services:
       - name: default
         evalWorker:
           enabled: true
         # existing session, memory and redis settings stay as they are
   ```

## Add a judge provider

1. Create a Provider for the judge model. The model ID is an example.

   ```yaml
   apiVersion: omnia.altairalabs.ai/v1alpha1
   kind: Provider
   metadata:
     name: claude-haiku
   spec:
     type: claude
     model: claude-haiku-4-5-20251001
     credential:
       secretRef:
         name: anthropic-api-key
   ```

2. Add it to the AgentRuntime's `providers` under a name of your choice:

   ```yaml
   spec:
     providers:
       - name: default
         providerRef:
           name: claude-sonnet
       - name: judge
         providerRef:
           name: claude-haiku
   ```

The worker builds a judge from every entry in `spec.providers`. An `llm_judge` eval picks one with `params.provider`. Without it, the worker uses the entry whose name sorts first alphabetically, which here is `default`, the agent's own model.

## Define evals in the PromptPack

Add the evals to the pack-level `evals` array in `pack.json`, or to `prompts.<name>.evals` for the prompt the agent opens at. The eval-worker runs the pack-level array plus that one prompt's evals, and a prompt-level eval replaces a pack-level eval with the same `id`. The prompt the agent opens at is the pack's only prompt, or `default` in a pack with several. Evals under any other prompt do not run in the worker.

```json
{
  "evals": [
    {
      "id": "helpfulness",
      "type": "llm_judge",
      "trigger": "every_turn",
      "params": {
        "provider": "judge",
        "criteria": "Is the response helpful, accurate and on-topic?",
        "rubric": "Give a high score only when the answer is correct and complete."
      }
    },
    {
      "id": "no-competitor-mentions",
      "type": "regex",
      "trigger": "every_turn",
      "params": {
        "pattern": "competitor-name",
        "expect_match": false
      }
    },
    {
      "id": "resolution-check",
      "type": "llm_judge",
      "trigger": "on_session_complete",
      "params": {
        "provider": "judge",
        "criteria": "Did the agent fully resolve the customer's issue?"
      }
    }
  ]
}
```

The [PromptPack reference](/reference/core/promptpack/#evals) lists the eval types, their parameters and the triggers. A session counts as complete when session-api marks it `completed` or when no assistant message arrives for 5 minutes.

PromptPacks are immutable once published. Apply the change as a new pack version.

To run an eval on a share of turns or sessions, use the `sample_turns` or `sample_sessions` trigger with `sample_percentage` on the eval. This limits judge cost.

To route an eval to a custom worker group, add `"groups": ["custom-safety"]` to the eval, then list that group under `evals.worker.groups`.

## Verify

1. Check that the worker Deployment exists and is running:

   ```bash
   kubectl get deploy -l app.kubernetes.io/name=arena-eval-worker
   kubectl logs -l app.kubernetes.io/component=eval-worker --tail=50
   ```

   Worker Deployments are named `arena-eval-worker-<service group>`.

2. Chat with the agent, then list recent results. Set `SESSION_API_URL` to the group's `sessionURL` in the Workspace's `status.services`. `namespace` is required, and the endpoint returns 400 without it.

   ```bash
   curl "$SESSION_API_URL/api/v1/eval-results?namespace=default&agentName=my-agent&limit=5"
   ```

   Each row has a `source` of `runtime-inline` or `worker`. Other endpoints: `GET /api/v1/sessions/SESSION_ID/eval-results`, and `GET /api/v1/sessions/SESSION_ID/eval-results/summary?namespace=default`. See the [session-api reference](/reference/platform/session-api/#eval-results).

3. In the dashboard, open a session from **Sessions**. The results appear in the **Checks** view of the session review. **Quality** in the sidebar shows score trends and a per-eval breakdown, filtered by time range, agent and PromptPack.

## Troubleshooting

### No eval-worker Deployment exists

The agent uses the PromptKit framework and its service group has no `evalWorker.enabled: true`. Add it as in step 2 of [Enable evals](#enable-evals).

### An llm_judge eval runs against the wrong model

The eval has no `params.provider`, so the worker uses the first provider alphabetically. Set `params.provider` to the name of the judge entry.

### An eval produces no rows

Check the worker log for these messages:

- `worker eval group filter matched no evals`: no eval for that trigger belongs to a group in `evals.worker.groups`.
- `evals skipped` with reason `noAssistantAnswer`: every assistant message in the session was empty.
- `evals skipped: session content suppressed`: the user opted out, so the session content was replaced.
- `no evals to run`: the session has no PromptPack.

## Related pages

- [Realtime evals](/explanation/evaluation/realtime-evals/) explains the inline and worker paths.
- [AgentRuntime reference](/reference/core/agentruntime/#evals) lists the `spec.evals` fields.
- [PromptPack reference](/reference/core/promptpack/#evals) lists eval types and triggers.