# Arena Fleet: Run Your First Evaluation

In this tutorial we run a prompt evaluation in your cluster. By the end, an ArenaRun has executed one scenario against a real LLM and reported a pass rate.

## What you will build

We create an ArenaProject that holds the bundle, a Provider that supplies the LLM, and an ArenaJob that names both. The ArenaJob is a template: it creates an ArenaRun, and the ArenaRun executes the evaluation. [Arena architecture](/explanation/evaluation/arena-fleet/) explains how the resources fit together.

```mermaid
flowchart LR
    ArenaProject -->|sourceRef| ArenaJob
    Provider -->|providerRef| ArenaJob
    ArenaJob -->|creates| ArenaRun
    ArenaRun --> Result["status.result"]
```

## Prerequisites

- A Kubernetes cluster with Omnia installed and `enterprise.enabled=true`
- `kubectl` configured to access the cluster
- An Anthropic API key

## Step 1: Create an ArenaProject

An ArenaProject is a git repository holding your bundle. We host it in the cluster and seed it from a ConfigMap. Each key in the ConfigMap becomes a file in the first commit, and `__` (double underscore) in a key stands for `/`.

Create a ConfigMap containing the whole bundle: the arena config file, one prompt, and one scenario with an assertion. Save it as `configmap.yaml`:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: greeting-bundle
  namespace: default
data:
  config.arena.yaml: |
    apiVersion: promptkit.altairalabs.ai/v1alpha1
    kind: Arena
    metadata:
      name: greeting-eval
    spec:
      prompt_configs:
        - id: assistant
          file: prompts/assistant.yaml
      # The ArenaJob supplies providers from Provider resources.
      providers: []
      scenarios:
        - file: scenarios/greeting.scenario.yaml
      defaults:
        temperature: 0.7
        max_tokens: 200
        output:
          dir: out
          formats:
            - json

  # "prompts__assistant.yaml" decodes to "prompts/assistant.yaml"
  prompts__assistant.yaml: |
    apiVersion: promptkit.altairalabs.ai/v1alpha1
    kind: PromptConfig
    metadata:
      name: assistant
    spec:
      task_type: assistant
      version: v1.0.0
      description: A friendly greeting assistant
      system_template: "You are a friendly assistant. Respond warmly to greetings."

  # "scenarios__greeting.scenario.yaml" decodes to "scenarios/greeting.scenario.yaml"
  scenarios__greeting.scenario.yaml: |
    apiVersion: promptkit.altairalabs.ai/v1alpha1
    kind: Scenario
    metadata:
      name: greeting-test
    spec:
      id: greeting-test
      task_type: assistant
      description: Greeting test
      turns:
        - role: user
          content: "Say hello to the world."
          assertions:
            - type: contains
              params:
                patterns:
                  - "hello"
```

Now create the ArenaProject. The ConfigMap becomes its first commit on `main`. Save it as `arenaproject.yaml`:

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaProject
metadata:
  name: greeting-source
  namespace: default
spec:
  repository:
    hosted:
      seed:
        configMap:
          name: greeting-bundle
```

Apply both resources:

```bash
kubectl apply -f configmap.yaml
kubectl apply -f arenaproject.yaml
```

Check that the project is ready:

```bash
kubectl get arenaproject greeting-source
```

You see:

```
NAME              READY   URL   PHASE   AGE
greeting-source   True          Ready   10s
```

The seed is applied once. `status.repository.seedCommit` holds the commit it made, and there is no `ArenaConfig` resource: the config is the `config.arena.yaml` file in the bundle. See the [arena config file reference](/reference/evaluation/arenaconfig/) for its schema.

## Step 2: Configure a Provider

The ArenaJob resolves its providers from Provider resources, so the bundle needs no provider file. Create the Secret that holds your API key:

```bash
kubectl create secret generic llm-credentials \
  --namespace default \
  --from-literal=ANTHROPIC_API_KEY=<your-anthropic-api-key>
```

Create the Provider. Save it as `provider.yaml`:

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: Provider
metadata:
  name: claude-provider
  namespace: default
spec:
  type: claude
  model: claude-sonnet-4-20250514
  credential:
    secretRef:
      name: llm-credentials
```

```bash
kubectl apply -f provider.yaml
```

Check that the Provider is ready:

```bash
kubectl get provider claude-provider
```

You see:

```
NAME              READY   TYPE     MODEL                      PHASE   AGE
claude-provider   True    claude   claude-sonnet-4-20250514   Ready   15s
```

## Step 3: Create an ArenaJob

The ArenaJob references the project, selects the config file with `arenaFile`, and maps the Provider into the config's `default` provider group. Save it as `arenajob.yaml`:

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
metadata:
  name: greeting-eval-001
  namespace: default
spec:
  sourceRef:
    name: greeting-source
  arenaFile: config.arena.yaml
  type: evaluation
  providers:
    default:
      - providerRef:
          name: claude-provider
```

```bash
kubectl apply -f arenajob.yaml
```

The ArenaJob runs nothing itself. It creates one ArenaRun, named after the job with a random suffix. Print the run's name:

```bash
kubectl get arenajob greeting-eval-001 -o jsonpath='{.status.lastRunRef.name}'
```

You see a name like:

```
greeting-eval-001-x7k2p
```

## Step 4: Watch the ArenaRun

Watch the run, using the name from the previous step:

```bash
kubectl get arenarun greeting-eval-001-x7k2p -w
```

When the run finishes, `PHASE` shows `Succeeded`:

```
NAME                      READY   JOB                 TRIGGER   PHASE       AGE
greeting-eval-001-x7k2p   True    greeting-eval-001   Manual    Succeeded   45s
```

The run records the commit it executed in `status.source.commit`. See [ArenaJob](/reference/evaluation/arenajob/#arenarun) for the run's phases.

## Step 5: View the results

Print the run's status:

```bash
kubectl get arenarun greeting-eval-001-x7k2p -o yaml
```

The `status` section contains:

```yaml
status:
  phase: Succeeded
  progress:
    completed: 1
    failed: 0
    pending: 0
    total: 1
  result:
    summary:
      failedItems: "0"
      passRate: "100.0"
      passedItems: "1"
      totalItems: "1"
```

The summary also holds `avgDurationMs` and a `details` JSON string with the per-scenario breakdown. Our scenario passed because the response contained "hello". See [ArenaJob](/reference/evaluation/arenajob/#result) for every summary key.

To read the worker's log, select its pod by the run's name:

```bash
kubectl logs -l omnia.altairalabs.ai/job=greeting-eval-001-x7k2p
```

## Next steps

- [Configure S3 storage](/how-to/evaluation/configure-arena-s3-storage/) to copy each run's output files to a bucket
- [Set up scheduled jobs](/how-to/evaluation/setup-arena-scheduled-jobs/) to run the evaluation on a cron schedule
- [Monitor Arena jobs](/how-to/evaluation/monitor-arena-jobs/) to follow runs as they execute
- [Work on an external repository](/how-to/evaluation/work-on-an-external-repository/) to run a project that lives on GitHub, GitLab or any https git host
- [ArenaJob providers](/reference/evaluation/arenajob/#providers) to test against several LLMs

## Cleanup

Remove the resources created in this tutorial. Deleting the ArenaJob deletes its ArenaRuns.

```bash
kubectl delete arenajob greeting-eval-001
kubectl delete arenaproject greeting-source
kubectl delete configmap greeting-bundle
kubectl delete provider claude-provider
kubectl delete secret llm-credentials
```