Arena Fleet: Run Your First Evaluation

In this tutorial we run a prompt evaluation in your cluster. By the end, an ArenaRun has executed one scenario against a real LLM and reported a pass rate.

What you will build

We create an ArenaProject that holds the bundle, a Provider that supplies the LLM, and an ArenaJob that names both. The ArenaJob is a template: it creates an ArenaRun, and the ArenaRun executes the evaluation. Arena architecture explains how the resources fit together.

flowchart LR
    ArenaProject -->|sourceRef| ArenaJob
    Provider -->|providerRef| ArenaJob
    ArenaJob -->|creates| ArenaRun
    ArenaRun --> Result["status.result"]

Prerequisites

• A Kubernetes cluster with Omnia installed and enterprise.enabled=true
• kubectl configured to access the cluster
• An Anthropic API key

Step 1: Create an ArenaProject

An ArenaProject is a git repository holding your bundle. We host it in the cluster and seed it from a ConfigMap. Each key in the ConfigMap becomes a file in the first commit, and __ (double underscore) in a key stands for /.

Create a ConfigMap containing the whole bundle: the arena config file, one prompt, and one scenario with an assertion. Save it as configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: greeting-bundle
  namespace: default
data:
  config.arena.yaml: |
    apiVersion: promptkit.altairalabs.ai/v1alpha1
    kind: Arena
    metadata:
      name: greeting-eval
    spec:
      prompt_configs:
        - id: assistant
          file: prompts/assistant.yaml
      # The ArenaJob supplies providers from Provider resources.
      providers: []
      scenarios:
        - file: scenarios/greeting.scenario.yaml
      defaults:
        temperature: 0.7
        max_tokens: 200
        output:
          dir: out
          formats:
            - json

  # "prompts__assistant.yaml" decodes to "prompts/assistant.yaml"
  prompts__assistant.yaml: |
    apiVersion: promptkit.altairalabs.ai/v1alpha1
    kind: PromptConfig
    metadata:
      name: assistant
    spec:
      task_type: assistant
      version: v1.0.0
      description: A friendly greeting assistant
      system_template: "You are a friendly assistant. Respond warmly to greetings."

  # "scenarios__greeting.scenario.yaml" decodes to "scenarios/greeting.scenario.yaml"
  scenarios__greeting.scenario.yaml: |
    apiVersion: promptkit.altairalabs.ai/v1alpha1
    kind: Scenario
    metadata:
      name: greeting-test
    spec:
      id: greeting-test
      task_type: assistant
      description: Greeting test
      turns:
        - role: user
          content: "Say hello to the world."
          assertions:
            - type: contains
              params:
                patterns:
                  - "hello"

Now create the ArenaProject. The ConfigMap becomes its first commit on main. Save it as arenaproject.yaml:

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaProject
metadata:
  name: greeting-source
  namespace: default
spec:
  repository:
    hosted:
      seed:
        configMap:
          name: greeting-bundle

Apply both resources:

kubectl apply -f configmap.yaml
kubectl apply -f arenaproject.yaml

Check that the project is ready:

kubectl get arenaproject greeting-source

You see:

NAME              READY   URL   PHASE   AGE
greeting-source   True          Ready   10s

The seed is applied once. status.repository.seedCommit holds the commit it made, and there is no ArenaConfig resource: the config is the config.arena.yaml file in the bundle. See the arena config file reference for its schema.

Step 2: Configure a Provider

The ArenaJob resolves its providers from Provider resources, so the bundle needs no provider file. Create the Secret that holds your API key:

kubectl create secret generic llm-credentials \
  --namespace default \
  --from-literal=ANTHROPIC_API_KEY=<your-anthropic-api-key>

Create the Provider. Save it as provider.yaml:

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: Provider
metadata:
  name: claude-provider
  namespace: default
spec:
  type: claude
  model: claude-sonnet-4-20250514
  credential:
    secretRef:
      name: llm-credentials

kubectl apply -f provider.yaml

Check that the Provider is ready:

kubectl get provider claude-provider

You see:

NAME              READY   TYPE     MODEL                      PHASE   AGE
claude-provider   True    claude   claude-sonnet-4-20250514   Ready   15s

Step 3: Create an ArenaJob

The ArenaJob references the project, selects the config file with arenaFile, and maps the Provider into the config’s default provider group. Save it as arenajob.yaml:

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
metadata:
  name: greeting-eval-001
  namespace: default
spec:
  sourceRef:
    name: greeting-source
  arenaFile: config.arena.yaml
  type: evaluation
  providers:
    default:
      - providerRef:
          name: claude-provider

kubectl apply -f arenajob.yaml

The ArenaJob runs nothing itself. It creates one ArenaRun, named after the job with a random suffix. Print the run’s name:

kubectl get arenajob greeting-eval-001 -o jsonpath='{.status.lastRunRef.name}'

You see a name like:

greeting-eval-001-x7k2p

Step 4: Watch the ArenaRun

Watch the run, using the name from the previous step:

kubectl get arenarun greeting-eval-001-x7k2p -w

When the run finishes, PHASE shows Succeeded:

NAME                      READY   JOB                 TRIGGER   PHASE       AGE
greeting-eval-001-x7k2p   True    greeting-eval-001   Manual    Succeeded   45s

The run records the commit it executed in status.source.commit. See ArenaJob for the run’s phases.

Step 5: View the results

Print the run’s status:

kubectl get arenarun greeting-eval-001-x7k2p -o yaml

The status section contains:

status:
  phase: Succeeded
  progress:
    completed: 1
    failed: 0
    pending: 0
    total: 1
  result:
    summary:
      failedItems: "0"
      passRate: "100.0"
      passedItems: "1"
      totalItems: "1"

The summary also holds avgDurationMs and a details JSON string with the per-scenario breakdown. Our scenario passed because the response contained “hello”. See ArenaJob for every summary key.

To read the worker’s log, select its pod by the run’s name:

kubectl logs -l omnia.altairalabs.ai/job=greeting-eval-001-x7k2p

Next steps

• Configure S3 storage to copy each run’s output files to a bucket
• Set up scheduled jobs to run the evaluation on a cron schedule
• Monitor Arena jobs to follow runs as they execute
• Work on an external repository to run a project that lives on GitHub, GitLab or any https git host
• ArenaJob providers to test against several LLMs

Cleanup

Remove the resources created in this tutorial. Deleting the ArenaJob deletes its ArenaRuns.

kubectl delete arenajob greeting-eval-001
kubectl delete arenaproject greeting-source
kubectl delete configmap greeting-bundle
kubectl delete provider claude-provider
kubectl delete secret llm-credentials