Configure realtime evals

Enterprise

Enable realtime evals on an AgentRuntime so that live conversations are scored against the eval definitions in its PromptPack.

Prerequisites

A session-api with PostgreSQL and Redis in the agent’s service group. session-api publishes session events to a Redis stream and stores results in the eval_results table.
A PromptPack that the agent already uses.
A Provider for the judge model, if you use llm_judge evals. The Provider’s role must be llm.

Enable evals

Set evals.enabled on the AgentRuntime:
```
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: AgentRuntime
metadata:
  name: my-agent
spec:
  evals:
    enabled: true
```
Without other settings, deterministic evals (contains, regex) run inside the runtime and LLM-judge and external evals run in the eval-worker. To change that split, set evals.inline.groups and evals.worker.groups. See the AgentRuntime reference for both fields and their defaults.

If the agent uses the PromptKit framework (the default) and the pack has llm_judge or other external evals, opt the service group into an eval-worker. The operator creates a worker for non-PromptKit agents without this step.

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: Workspace
spec:
  services:
    - name: default
      evalWorker:
        enabled: true
      # existing session, memory and redis settings stay as they are

Add a judge provider

Create a Provider for the judge model. The model ID is an example.

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: Provider
metadata:
  name: claude-haiku
spec:
  type: claude
  model: claude-haiku-4-5-20251001
  credential:
    secretRef:
      name: anthropic-api-key

Add it to the AgentRuntime’s providers under a name of your choice:

spec:
  providers:
    - name: default
      providerRef:
        name: claude-sonnet
    - name: judge
      providerRef:
        name: claude-haiku

The worker builds a judge from every entry in spec.providers. An llm_judge eval picks one with params.provider. Without it, the worker uses the entry whose name sorts first alphabetically, which here is default, the agent’s own model.

Define evals in the PromptPack

Add the evals to the pack-level evals array in pack.json, or to prompts.<name>.evals for the prompt the agent opens at. The eval-worker runs the pack-level array plus that one prompt’s evals, and a prompt-level eval replaces a pack-level eval with the same id. The prompt the agent opens at is the pack’s only prompt, or default in a pack with several. Evals under any other prompt do not run in the worker.

{
  "evals": [
    {
      "id": "helpfulness",
      "type": "llm_judge",
      "trigger": "every_turn",
      "params": {
        "provider": "judge",
        "criteria": "Is the response helpful, accurate and on-topic?",
        "rubric": "Give a high score only when the answer is correct and complete."
      }
    },
    {
      "id": "no-competitor-mentions",
      "type": "regex",
      "trigger": "every_turn",
      "params": {
        "pattern": "competitor-name",
        "expect_match": false
      }
    },
    {
      "id": "resolution-check",
      "type": "llm_judge",
      "trigger": "on_session_complete",
      "params": {
        "provider": "judge",
        "criteria": "Did the agent fully resolve the customer's issue?"
      }
    }
  ]
}

The PromptPack reference lists the eval types, their parameters and the triggers. A session counts as complete when session-api marks it completed or when no assistant message arrives for 5 minutes.

PromptPacks are immutable once published. Apply the change as a new pack version.

To run an eval on a share of turns or sessions, use the sample_turns or sample_sessions trigger with sample_percentage on the eval. This limits judge cost.

To route an eval to a custom worker group, add "groups": ["custom-safety"] to the eval, then list that group under evals.worker.groups.

Verify

Check that the worker Deployment exists and is running:

kubectl get deploy -l app.kubernetes.io/name=arena-eval-worker
kubectl logs -l app.kubernetes.io/component=eval-worker --tail=50

Worker Deployments are named arena-eval-worker-<service group>.

Chat with the agent, then list recent results. Set SESSION_API_URL to the group’s sessionURL in the Workspace’s status.services. namespace is required, and the endpoint returns 400 without it.
Terminal window
```
curl "$SESSION_API_URL/api/v1/eval-results?namespace=default&agentName=my-agent&limit=5"
```
Each row has a source of runtime-inline or worker. Other endpoints: GET /api/v1/sessions/SESSION_ID/eval-results, and GET /api/v1/sessions/SESSION_ID/eval-results/summary?namespace=default. See the session-api reference.
In the dashboard, open a session from Sessions. The results appear in the Checks view of the session review. Quality in the sidebar shows score trends and a per-eval breakdown, filtered by time range, agent and PromptPack.

Troubleshooting

No eval-worker Deployment exists

The agent uses the PromptKit framework and its service group has no evalWorker.enabled: true. Add it as in step 2 of Enable evals.

An llm_judge eval runs against the wrong model

The eval has no params.provider, so the worker uses the first provider alphabetically. Set params.provider to the name of the judge entry.

An eval produces no rows

Check the worker log for these messages:

worker eval group filter matched no evals: no eval for that trigger belongs to a group in evals.worker.groups.
evals skipped with reason noAssistantAnswer: every assistant message in the session was empty.
evals skipped: session content suppressed: the user opted out, so the session content was replaced.
no evals to run: the session has no PromptPack.

Realtime evals explains the inline and worker paths.
AgentRuntime reference lists the spec.evals fields.
PromptPack reference lists eval types and triggers.