# ArenaJob CRD

:::note[Enterprise Feature]
ArenaJob is an enterprise feature. The Arena CRDs come from the `omnia-crds` chart, which installs them by default; the Arena controllers run only when `enterprise.enabled=true` in the app chart's values. See [Installing a License](/how-to/operations/install-license/) for details.
:::

The ArenaJob custom resource is a **template** for test executions that run scenarios from an [ArenaProject](/reference/evaluation/arenaproject/) bundle. It reads the [arena config file](/reference/evaluation/arenaconfig/) inside that bundle (selected by `spec.arenaFile`) and describes evaluation, load testing, or data generation work with configurable workers and output destinations.

An ArenaJob does not execute anything itself. Each execution is an **ArenaRun** (short name `arun`) that the job creates, owns and names `<job-name>-<suffix>`: once when the job is first seen if it has no `schedule`, or on every cron tick if it has one. The run carries a frozen copy of the job's spec (`spec.template`), so editing the job never changes what an existing run did. Execution state (phase, progress, results, timings) lives on the ArenaRun; the ArenaJob's status describes scheduling only. See [ArenaRun](#arenarun) below.

## API Version

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
```

**Short name:** `aj` (e.g. `kubectl get aj`).

## Spec Fields

### `sourceRef`

Reference to the ArenaProject, in the job's namespace, whose repository holds the arena config file, scenarios and prompts. Required. A run whose project does not exist yet, or is not `Ready`, stays `Pending` and is retried every 30 seconds.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | string | Yes | Name of the ArenaProject |

```yaml
spec:
  sourceRef:
    name: my-evaluation-source
```

### `ref`

The project ref a run executes: a branch, a tag, a full ref name or a full commit SHA. Defaults to the project's default branch (`main` for a hosted project). See [Refs](/reference/evaluation/arenaproject/#refs) for what is accepted.

Each run resolves the ref to a commit SHA when it starts and records it on the ArenaRun's `status.source`. A job naming a branch runs whatever the branch holds when each run starts; a job naming a SHA runs that commit every time. A ref that names nothing in the repository fails the run with reason `RefNotFound`, and a commit that cannot be exported (see [ArenaProject snapshots](/reference/evaluation/arenaproject/#snapshots)) fails it with `SnapshotUnexportable`.

```yaml
spec:
  sourceRef:
    name: my-evaluation-source
  ref: refs/tags/v1.2.0
```

### `arenaFile`

Path to the arena config file **within the project's repository**, matched exactly against the bundle contents. Defaults to `config.arena.yaml`.

It names one file: a run loads exactly one arena config. A value containing a
glob character (`*`, `?` or `[`), such as `evals/*.arena.yaml`, is rejected
when the ArenaJob is applied.

See the [Arena Config File](/reference/evaluation/arenaconfig/) reference for the schema of this file.

```yaml
spec:
  arenaFile: config.arena.yaml
```

### `type`

The type of job to execute. Defaults to `evaluation`.

| Value | Description |
|-------|-------------|
| `evaluation` | Run prompt evaluation against test scenarios (default) |
| `loadtest` | Run load testing against providers |
| `datagen` | Generate synthetic data using prompts |

```yaml
spec:
  type: evaluation
```

### `trials`

Number of times to repeat each scenario × provider combination. Overrides per-scenario trials defined in the scenario YAML files. Minimum `1`.

- For **evaluation** jobs, trials provide statistical confidence (pass rate, flakiness score).
- For **loadtest** jobs, trials define the total load volume, consumed under concurrency control.

```yaml
spec:
  trials: 20
```

### `scenarios`

Filter which scenarios to run from the arena file. If not specified, runs all scenarios defined in the arena file.

| Field | Type | Description |
|-------|------|-------------|
| `include` | []string | Glob patterns for scenarios to include |
| `exclude` | []string | Glob patterns for scenarios to exclude |

Exclusions are applied after inclusions. If `include` is empty, all scenarios are included by default.

```yaml
spec:
  scenarios:
    include:
      - "scenarios/critical-*.yaml"
    exclude:
      - "*-slow.yaml"
```

### `budgetLimit`

The most this job may spend, in `budgetCurrency`, before it is stopped.

```yaml
spec:
  budgetLimit: "25.00"
  budgetCurrency: "USD"
```

It applies to any run (an evaluation, a data-generation run, or a load test), because the cap is enforced on one accumulator holding the run's whole spend. The increment that crosses the limit marks the run stopped and discards every queued item no worker has started; the workers find an empty queue and finish, and the ArenaRun fails with reason `BudgetExceeded` (a Warning event on the run, and `budgetBreached`, `totalCost` and `drainedItems` in its `status.result.summary`).

:::caution
`spec.loadTest.budgetLimit` has been removed. A job that still sets it runs with no cap at all: the field is pruned from an unknown schema rather than rejected, so nothing warns you. Move the value to `spec.budgetLimit`.
:::

The value must be a bare decimal, `^[0-9]+(\.[0-9]+)?$`. A currency symbol, a sign or an exponent is rejected by the API server on apply:

```
The ArenaJob "x" is invalid: spec.budgetLimit: Invalid value: "$25.00":
spec.budgetLimit in body should match '^[0-9]+(\.[0-9]+)?$'
```

The admission webhook checks the same thing, but the pattern is what you can rely on: `webhook.enabled` defaults to `false`, so in a default install the webhook is not running and the schema is the only validation there is.

The cap is a ceiling rather than an exact stop. Cost reaches the run's accumulator as each request completes (through the queue for a run that drains one, and charged directly by a load test with a `duration`), so the overshoot is the requests already in flight when the cap is crossed (at most `concurrency` requests' worth). Every worker learns the cap is spent at its next completion and starts nothing further. The cap binds on the run's total; it is not divided between workers.

### `loadTest`

Settings specific to load testing jobs (used when `type: loadtest`).

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `concurrency` | integer | 1 | Maximum number of work items in flight across all workers. The ceiling is divided evenly across the worker pods, rounded up, and each pod enforces its share without reading the others' counts. Minimum 1. |
| `vusPerWorker` | integer | 1 | Number of virtual users (concurrent goroutines) per worker pod. Each VU independently pops, executes, and reports work items. Minimum 1. |
| `duration` | string | - | How long to apply load for (e.g. `"10m"`). **Setting it changes where the work comes from** — see below. |
| `rate` | integer | - | Target arrivals per second across all workers. Selects the open model; requires `duration`. Minimum 1. |
| `ramp` | object | - | Linear concurrency ramp-up / ramp-down (see below). |
| `thresholds` | []object | - | SLO targets evaluated after the load test completes (see below). |

#### `loadTest.duration`: a clock instead of a matrix

Without `duration`, a load test's volume is `scenarios × providers × trials`, and its length is `items × latency ÷ concurrency`, which varies because latency is also what the test measures.

With `duration`, no matrix is queued. Each worker generates from the scenario × provider pool until the clock runs out, and `trials` is ignored. `status.result.summary.timeSeries` carries per-second rps, error rate and latency percentiles.

#### `loadTest.rate`: offered arrivals instead of a ceiling

`rate` switches the run from a closed model to an open one:

| | driver | `concurrency` is |
|---|---|---|
| no `rate` | as fast as the ceiling allows | the target |
| `rate` set | an arrival schedule | a safety ceiling |

Arrivals the workers cannot keep up with are reported, never queued. Gate on it with the `arrival_lag` threshold.

```yaml
spec:
  type: loadtest
  loadTest:
    duration: 10m       # run for ten minutes, not for N items
    rate: 20            # offer twenty arrivals a second
    concurrency: 50     # now a safety limit, not the driver
    thresholds:
      - metric: arrival_lag
        operator: "<"
        value: "0.02"   # fail if 2% of arrivals were missed
```

#### `loadTest.ramp`

Controls how concurrency changes over the course of a load test. Both values are duration strings (e.g. `"2m"`, `"30s"`).

| Field | Type | Description |
|-------|------|-------------|
| `up` | string | Duration to linearly ramp from 0 to target concurrency at the start. |
| `down` | string | Duration to linearly ramp from target concurrency to 1 at the end. With `loadTest.duration` set, the ramp-down occupies the last `down` of the run. Without it, the ramp-down starts once a worker's last completed item reports fewer than `concurrency × 2` items still pending. |

#### `loadTest.thresholds`

Each threshold is an SLO gate evaluated after the load test completes. The job fails if any threshold is violated.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `metric` | string | Yes | Metric to evaluate (see allowed values below). |
| `operator` | string | Yes | Comparison operator: `<`, `>`, `<=`, `>=`. |
| `value` | string | Yes | Target value to compare against. |

**Allowed `metric` values:** `latency_avg`, `latency_p50`, `latency_p90`, `latency_p95`, `latency_p99`, `ttft_avg`, `ttft_p50`, `ttft_p90`, `ttft_p95`, `ttft_p99`, `error_rate`, `pass_rate`, `total_cost`, `rate_limit_rate`, `arrival_lag`.

| Metric | Definition |
|--------|------------|
| `latency_avg` | Mean item latency, from the run's counters. Available once one item has completed. |
| `latency_p50`, `_p90`, `_p95`, `_p99` | Percentile of the run's latency distribution. |
| `ttft_avg`, `ttft_p50`, `_p90`, `_p95`, `_p99` | Mean or percentile of time-to-first-token, over the items that reported one. |
| `error_rate` | Failed items ÷ total items. |
| `pass_rate` | Passed items ÷ total items. |
| `rate_limit_rate` | Items whose error text matches a rate-limit phrase (for example "rate limit", "too many requests", "quota exceeded", "resource exhausted", "overloaded") or contains a standalone `429`, ÷ total items. |
| `arrival_lag` | Scheduled arrivals the generator could not keep up with ÷ arrivals offered. |
| `total_cost` | Accumulated cost, in `budgetCurrency`. |

An item's latency is the time the system under test spent answering it: the sum of its provider calls (for an `agentRef`, each call is one round trip to the agent, so the agent's own tool calls are inside it). The worker's own time (reading the run, building the engine, recording the session, running assertions) is not part of it, so a load test measures the target rather than itself. Nor are tools the worker executes itself (a scenario's mock and HTTP tools) or the self-play user simulator's provider calls: in a multi-turn self-play conversation, only the assistant turns are timed. When no provider call reports a duration, the engine's execution time is used instead. The whole item, overhead included, is recorded separately on each item result as the `itemDurationMs` metric.

Latency percentiles are answered from the run's latency distribution. A load test keeps one when it has a `duration`, or when it drains a matrix with `concurrency` or `vusPerWorker` above 1. An enumerated load test left at the defaults (`concurrency: 1`, `vusPerWorker: 1`) runs sequentially, keeps no distribution, and its percentile rules report `unavailable`.

`arrival_lag` only means anything for a run given a `rate`: a run with no arrival schedule reports `unavailable` rather than a zero, because a paced run that kept up also reports zero, and the two are different facts. A metric nobody could measure reports `unavailable` and passes.

`ttft_*` needs a provider that reports a first token, meaning a fleet entry (`agentRef`): the time is read off the agent's stream after execution. It also needs a distribution, on the same terms as the latency percentiles above. A raw `providerRef` run measures total duration only. TTFT is kept as its own distribution rather than folded into latency's, so a run that measured no first token reports `unavailable` instead of a wrong total-duration number.

**`value` formats:**
- Latency / TTFT metrics: a duration string (e.g. `"3s"`, `"500ms"`).
- Rate metrics (`error_rate`, `pass_rate`, `rate_limit_rate`, `arrival_lag`): a float string between 0 and 1 (e.g. `"0.01"`, `"0.95"`). A target outside that range is rejected: `error_rate < 50` would otherwise parse and pass every run.
- Cost metric (`total_cost`): a numeric string (e.g. `"50.00"`).

The ArenaRun records each result in `status.result.summary` as `threshold:<metric>` (for example `3.20s < 3s FAIL`; `unavailable` or `parse error` when the rule could not be judged) plus `thresholds_passed` (for example `3/4 passed`). A violated threshold moves the run to `Failed` with reason `TestsFailed`. An enumerated load test also fails when any item fails; a run with a `duration` fails only on its thresholds.

When the admission webhook is running (`webhook.enabled: true`; it defaults to `false`), a `value` in none of these forms is rejected on apply, as are an unparseable `duration` or `ramp` value and a `rate` without a `duration`. Whether or not the webhook caught it, a `value` that cannot be read is reported as `parse error` at evaluation and fails the run rather than passing it.

```yaml
spec:
  type: loadtest
  budgetLimit: "25.00"
  budgetCurrency: USD
  loadTest:
    concurrency: 50
    vusPerWorker: 10
    ramp:
      up: 2m
      down: 30s
    thresholds:
      - metric: latency_p95
        operator: "<"
        value: "3s"
      - metric: error_rate
        operator: "<="
        value: "0.01"
```

### `dataGen`

Settings specific to data generation jobs (used when `type: datagen`).

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `count` | integer | 100 | Number of items to generate |
| `format` | string | "jsonl" | Output format: `jsonl` or `json` |

```yaml
spec:
  type: datagen
  dataGen:
    count: 1000
    format: jsonl
```

`csv` is rejected at admission: a conversation is a nested list of turns, and flattening it into columns loses that structure.

#### Where the generated records go

Each generated conversation lands in two places, holding the same content:

- Sessions: recorded as an ordinary session, tagged `kind:corpus` and `arena-run:<run name>`. This is what the dashboard's Corpus tab reads, and what you query: `GET /api/v1/sessions?tag=kind:corpus&tag=arena-run:<run>`. A datagen run records these without `sessionRecording` being set; see [`sessionRecording`](#sessionrecording).
- A file, in `format`, written to the run's `output` target: the export, downloaded or fed to training, written whether or not sessions were recorded.

There is no separate corpus store to configure.

### `workers`

Configure the worker pool for job execution.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `replicas` | integer | 1 | Number of worker replicas |
| `podOverrides` | object | - | Customizes the worker Job Pods (scheduling, ServiceAccount, CSI secret-stores, custom `envFrom` for provider credentials, etc.). See [`podOverrides`](/reference/core/agentruntime/#podoverrides) |

```yaml
spec:
  workers:
    replicas: 10
```

`replicas` sets the worker Job's `parallelism` and `completions`: that many worker pods run each ArenaRun's work concurrently.

A workspace's [`spec.quotas.arena.maxWorkersPerJob`](/reference/core/workspace/#quotasarena) caps it: a run asking for more runs at the cap, and its `status.workers` records the count it started with. With the licence webhooks enabled, an explicit `replicas` above the cap is refused on create, or when an update raises it, and the licence's worker-replica limit is checked against the capped count, as the controller checks it.

### `providers`

Maps group names to provider groups. Group names correspond to the arena config file's provider groups — the `group:` value on each `providers:` entry in `config.arena.yaml` (e.g. `"default"`, `"judge"`).

When `providers` is set, provider YAML files from the arena bundle are **ignored** and the worker resolves providers directly from CRDs.

Each entry is an `ArenaProviderEntry`, with one of the following fields set:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `providerRef` | object | Conditional | Reference to a Provider CRD |
| `providerRef.name` | string | Yes | Name of the Provider resource |
| `providerRef.namespace` | string | No | If set, must equal the ArenaJob's own namespace; a Provider in any other namespace is refused: the run's `JobCreated` condition goes `False` with reason `JobCreationFailed` and the controller retries with backoff |
| `agentRef` | object | Conditional | Reference to an AgentRuntime CRD |
| `agentRef.name` | string | Yes | Name of the AgentRuntime resource |

:::caution
Each entry must set exactly one of `providerRef` or `agentRef`. The `providers` map is excluded from CRD schema validation, so the API server does not check it. The admission webhook rejects a bad entry when `webhook.enabled` is `true`. Otherwise the ArenaRun controller refuses it: the run moves to `Failed` with reason `InvalidProviderEntry`, and the message names the group and the entry's index (array mode) or key (map mode).
:::

Agents and LLM providers are interchangeable in the scenario × provider matrix. An `agentRef` entry causes the worker to connect to the agent over WebSocket instead of making direct LLM API calls.

#### Array mode vs map mode

Each provider group accepts **two shapes**, and the two can be mixed across groups within the same job:

- **Array mode (default)** — a list of entries. The group is a *pool* of test providers, and every entry is run against every selected scenario.
- **Map mode** — an object keyed by config-provider-ID. Each key is the exact provider ID the arena config expects, mapped 1:1 to a CRD entry. Use this when the arena config references specific provider IDs (for example, the deploy wizard emits map mode so the config's provider IDs resolve exactly). Map-mode groups (judges, self-play roles) do not add entries to the scenario × provider matrix.

**Array mode:**

```yaml
spec:
  providers:
    default:
      - providerRef:
          name: gpt4-prod
      - providerRef:
          name: claude-sonnet
```

**Map mode:**

```yaml
spec:
  providers:
    default:
      # config-provider-ID → CRD
      primary:
        providerRef:
          name: gpt4-prod
      secondary:
        providerRef:
          name: claude-sonnet
```

#### Example: Separate Judge Provider

Use a dedicated provider group for the judge (evaluator) model. The group name (`judge`) must match a `group:` value used in the arena config's `providers:` list:

```yaml
spec:
  providers:
    default:
      - providerRef:
          name: gpt4-prod
      - providerRef:
          name: claude-sonnet
    judge:
      - providerRef:
          name: claude-opus
```

#### Example: Agent Entry

Reference a deployed AgentRuntime instead of a raw LLM provider. The worker connects to the agent's WebSocket endpoint:

```yaml
spec:
  providers:
    default:
      - agentRef:
          name: my-support-agent
```

#### Example: Self-Play with Mixed Types

Mix LLM providers and agents in a self-play evaluation:

```yaml
spec:
  providers:
    selfplay:
      - providerRef:
          name: gpt4-prod
      - agentRef:
          name: my-agent-v2
    judge:
      - providerRef:
          name: claude-opus
```

### `toolRegistries`

List of ToolRegistry CRD references, in the job's namespace, whose discovered tools override how the arena config's tools are executed. The tool YAML files in the bundle still define the tools; a tool whose name matches a discovered tool is rewritten to call that tool's endpoint over HTTP.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | string | Yes | Name of the ToolRegistry resource |

```yaml
spec:
  toolRegistries:
    - name: production-tools
```

#### How Tool Registries Work

1. The worker reads each referenced ToolRegistry CRD
2. It collects the tools listed in each registry's `status.discoveredTools`
3. For every tool file in the arena config whose tool name matches a discovered tool, the worker switches the tool to an HTTP executor at the discovered endpoint (method `POST` unless the file sets one) and takes the registry's description
4. Tools with no matching discovered tool, and `mcp_servers`, are used as written

#### Example: Multiple Tool Registries

```yaml
spec:
  toolRegistries:
    - name: core-tools
    - name: billing-tools
```

#### Combining Providers and Tool Registries

Example combining `providers` and `toolRegistries`:

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
metadata:
  name: production-eval
spec:
  sourceRef:
    name: my-source
  arenaFile: config.arena.yaml
  providers:
    default:
      - providerRef:
          name: gpt4-prod
      - providerRef:
          name: claude-sonnet
    judge:
      - providerRef:
          name: claude-opus
  toolRegistries:
    - name: production-tools
  workers:
    replicas: 5
  output:
    type: s3
    s3:
      bucket: arena-results
      prefix: "evals/"
```

### `output`

Configure where job results are stored.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | string | Yes | Destination type: s3, pvc |
| `s3` | object | Conditional | S3 configuration (when type is s3) |
| `pvc` | object | Conditional | PVC configuration (when type is pvc) |

#### S3 Output

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `bucket` | string | Yes | S3 bucket name |
| `prefix` | string | No | Key prefix for objects |
| `region` | string | No | AWS region |
| `endpoint` | string | No | Custom S3-compatible endpoint |
| `secretRef` | object | No | Credentials secret reference (keys: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) |

```yaml
spec:
  output:
    type: s3
    s3:
      bucket: arena-results
      prefix: "evals/nightly/"
      region: us-west-2
      secretRef:
        name: s3-credentials
```

#### PVC Output

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `claimName` | string | Yes | PVC name |
| `subPath` | string | No | Subdirectory within PVC |

```yaml
spec:
  output:
    type: pvc
    pvc:
      claimName: arena-results-pvc
      subPath: "evals/"
```

#### Without `output`

A run whose job sets no `output` keeps its engine output on the workspace content volume, in its own directory `arena/runs/{run}/`. The output holds the per-item JSON results, so you have them whether or not the run records sessions to session-api.

| Behaviour | Detail |
|---|---|
| When | The run's content comes from the workspace content volume, which is the case for every project run when the Arena controller has `--workspace-content-path` set. Otherwise the worker writes to its scratch space, and the output is gone when the pod exits |
| Worker mount | The content volume, read-write, at `/arena-output`, with a `subPath` of the run's directory alone. The project content stays a separate read-only mount |
| Run marker | The annotation `arena.omnia.altairalabs.ai/output: arena/runs/{run}` and the finalizer `arena.omnia.altairalabs.ai/output` |
| Download | `GET /api/v1/workspaces/{name}/arena/runs/{runName}/output` returns the directory as a zip, for a workspace viewer. The run page offers it as **Download output** once the run has finished or been cancelled |
| Removal | Deleting the run removes the directory. This includes runs pruned by `successfulRunsHistoryLimit` and `failedRunsHistoryLimit` |
| Failure | If the Arena controller cannot create the directory or mark the run, it logs the error and the worker writes to its scratch space. The run proceeds either way |
| No content volume | An Arena controller without `--workspace-content-path` adds no finalizer. One that has the finalizer on a run but no volume mounted logs and releases the run, leaving the directory |

The Arena controller removes the finalizer. If the controller is gone, for
example after Enterprise is disabled or the Arena controller is uninstalled,
a deleted run waits with a deletion timestamp. Remove the finalizer by hand,
then delete the run's directory from the volume yourself if you want it
gone:

```bash
kubectl patch arenarun <run> -n <namespace> --context <context> --type=json \
  -p '[{"op":"remove","path":"/metadata/finalizers/<index>"}]'
```

`<index>` is the position of `arena.omnia.altairalabs.ai/output` in the run's
`metadata.finalizers`, counted from 0:

```bash
kubectl get arenarun <run> -n <namespace> --context <context> \
  -o jsonpath='{.metadata.finalizers}'
```

When it is the run's only finalizer, a merge patch clears it without the
index:

```bash
kubectl patch arenarun <run> -n <namespace> --context <context> --type=merge \
  -p '{"metadata":{"finalizers":null}}'
```

### `schedule`

Configure scheduled/recurring execution. Each due tick creates a new ArenaRun (trigger `Schedule`). Without a `schedule` (or with an empty `cron`), the job creates exactly one run (trigger `Manual`) when it is first reconciled.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `cron` | string | - | Cron expression for scheduling |
| `timezone` | string | "UTC" | Timezone for cron |
| `concurrencyPolicy` | string | "Forbid" | `Allow` starts the new run alongside any still going; `Forbid` skips the tick (recording it as passed, with a `RunSkipped` event) while a run is active; `Replace` cancels the active runs (sets their `spec.cancelled`) and then starts the new one |

```yaml
spec:
  schedule:
    cron: "0 2 * * *"  # 2am daily
    timezone: "America/New_York"
    concurrencyPolicy: Forbid
```

Under a workspace [`spec.quotas.arena.maxJobsPerDay`](/reference/core/workspace/#quotasarena), a tick is also skipped, recorded as passed with a Warning `RunSkipped` event, when the runs already started or waiting in the last 24 hours use up the allowance. A run created by a tick that the workspace's `maxConcurrentJobs` cannot admit yet waits `Pending` instead; under `Forbid` that waiting run is active, so later ticks are skipped until it finishes.

### `suspend`

Stops the job starting new runs, without deleting it. Runs already going are left alone — to stop one, set `spec.cancelled` on the [ArenaRun](#arenarun). While suspended, `status.nextScheduleTime` is cleared and a `Suspended` event is recorded. Default `false`.

```yaml
spec:
  suspend: true
```

### `successfulRunsHistoryLimit` / `failedRunsHistoryLimit`

How many finished ArenaRuns the job keeps. Older ones beyond the limit are deleted. Defaults: `3` successful, `1` failed (a `Cancelled` run counts as failed). Minimum `0`. The per-scenario detail of every run lives in session-api, so pruning a run object does not lose its results.

```yaml
spec:
  successfulRunsHistoryLimit: 7
  failedRunsHistoryLimit: 3
```

### `serviceGroup`

The Workspace service group whose session-api this job's runs record into, and whose `redis`, when it sets one, holds the runs' work queue (see the ArenaRun's [`status.queue`](#queue)). For the session-api, it is resolved only when `sessionRecording` is `true`, or the job is a `datagen` job. For the queue, it is resolved whenever any group in the workspace sets a `redis`; a name the workspace does not declare then fails the run with `ServiceGroupNotFound`. When omitted, a workspace with exactly one service group resolves to it. A workspace with several is ambiguous: the controller does not pick one, and the run moves to `Failed` with reason `ServiceGroupAmbiguous`. A group the workspace does not have, a workspace with no service groups in its status, or a group that declares no session-api, leaves the worker Job uncreated: the run's `JobCreated` condition goes `False` with reason `JobCreationFailed` and the controller retries with backoff. Lowercase DNS label, at most 63 characters.

```yaml
spec:
  serviceGroup: default
```

### `verbose`

Enables verbose/debug logging for arena execution. When enabled, the worker pods run with `LOG_LEVEL=debug` and `ARENA_VERBOSE=true`, which turns on the arena engine's and PromptKit SDK's verbose logging.

```yaml
spec:
  verbose: true
```

### `sessionRecording`

Enables writing session data to session-api during execution. When `false` (default), no sessions are created and no events are recorded, reducing session-api load during high-volume load tests. Telemetry and traces are unaffected.

A `datagen` run records sessions whether or not this is set, because its sessions are its corpus. If the workspace has no session-api to record into and the field is not set, the run still writes its export, and emits a `CorpusNotRecorded` warning event; with the field set to `true`, the same problem stops the run, as for any other type.

When it is `true`, a load-test conversation against an `agentRef` entry is recorded by the agent's facade as its own session, and the worker adds arena context and any self-play or judge calls to that session. A conversation against a `providerRef` entry is recorded by the worker's own session manager.

```yaml
spec:
  sessionRecording: true
```

### `ttlSecondsAfterFinished`

How long Kubernetes keeps each run's finished worker Job (and its pods) before deleting it. Sets the worker Job's `ttlSecondsAfterFinished`; defaults to `3600` (one hour) when unset. It does not delete ArenaJobs or ArenaRuns — run history is bounded by the history limits above.

```yaml
spec:
  ttlSecondsAfterFinished: 86400  # 24 hours
```

## Status Fields

An ArenaJob's status describes scheduling. Per-execution state is on each [ArenaRun](#arenarun).

| Field | Description |
|-------|-------------|
| `active` | References to the ArenaRuns that have not finished |
| `lastRunRef` | Reference to the most recently created ArenaRun |
| `lastScheduleTime` | The last cron tick that was acted on (a run started or skipped) |
| `nextScheduleTime` | The next cron tick; cleared while `suspend` is `true` |
| `observedGeneration` | Most recent generation observed by the controller |

### `conditions`

| Type | Description |
|------|-------------|
| `Ready` | `True` (reason `Scheduled`) while the job can schedule runs; `False` with reason `InvalidSchedule` when the cron expression cannot be parsed |
| `Stalled` | `True` when the job cannot make progress without a spec change (an invalid schedule); `False` otherwise |

## ArenaRun

An ArenaRun (short name `arun`) is one execution of an ArenaJob. The job's controller creates it; you do not normally write one yourself.

| Spec field | Description |
|------------|-------------|
| `jobRef.name` | The ArenaJob this run was created from |
| `trigger` | `Schedule` (a cron tick) or `Manual` (default) |
| `template` | The ArenaJob `spec` frozen when the run was created. Every field documented above is read from here |
| `cancelled` | Set to `true` to stop the run: the controller deletes the worker Job, drains the run's remaining queued work, and moves the run to `Cancelled`. Has no effect once the run is `Succeeded`, `Failed` or `Cancelled` |

```bash
kubectl -n arena patch arenarun nightly-eval-x7k2p \
  --type=merge -p '{"spec":{"cancelled":true}}'
```

### `phase`

| Value | Description |
|-------|-------------|
| `Pending` | Waiting to start: the ArenaProject is not `Ready` yet, or the workspace's [`spec.quotas.arena`](/reference/core/workspace/#quotasarena) has not admitted the run (`WithinWorkspaceQuota=False`) |
| `Running` | The worker Job has been created and is executing |
| `Succeeded` | Completed successfully |
| `Failed` | Failed, including a ref that names nothing in the project (reason `RefNotFound`), a commit that cannot be exported (`SnapshotUnexportable`), a provider entry that sets both or neither of `providerRef` and `agentRef` (`InvalidProviderEntry`), an ambiguous service group (`ServiceGroupAmbiguous`), a named service group the workspace does not declare when its groups set their own Redis (`ServiceGroupNotFound`), a budget stop, a failed threshold, a licence violation or a work matrix over the item cap |
| `Cancelled` | Stopped by `spec.cancelled` |

### `source`

The commit the run executes, set once when the run resolves `template.ref` and never changed.

| Field | Description |
|-------|-------------|
| `repository` | The project's repository: its URL for an external project, `hosted` for a hosted one |
| `ref` | The ref the run asked for; the project's default branch when the template named none. A run started from the project editor (or the operator's start-run route) pins `spec.ref` to a SHA and shows the ref the person asked for here, from the job's `arena.omnia.altairalabs.ai/requested-ref` annotation |
| `commit` | The commit SHA `ref` resolved to |
| `contentPath` | The commit's snapshot, relative to the workspace content subtree (`arena/sources/<project>/.arena/versions/<commit>`), which the workers mount |

### `queue`

The Redis the run's work queue lives on, set once before the worker Job is created and never changed. See [`services[].redis`](/reference/core/workspace/#servicesredis) for how it is resolved.

| Field | Description |
|-------|-------------|
| `source` | `ServiceGroup` when the run's service group sets a `redis`; `Default` for the operator-wide arena queue (`enterprise.arena.queue.redis`) |
| `serviceGroup` | The service group the run resolved to: `template.serviceGroup`, or the workspace's only group. Empty when no group applies |
| `secretRef` | `name` and `key` of the Secret, in the run's namespace, holding the queue's Redis URL. Set for `ServiceGroup` only. It is the group's own `existingSecret`, or `<run>-queue-redis` (key `url`), which the controller writes for a `url`, `host` or `serviceRef` group and which is owned by the run |
| `configHash` | A fingerprint of the group's `redis` when the run started. If the group's `redis` later differs, the run gets the `QueueRedisDrifted` condition and stays on its queue |

A run that was already started when this field arrived is pinned to `Default`.

### `progress`

| Field | Description |
|-------|-------------|
| `total` | Total number of work items |
| `completed` | Successfully completed items |
| `failed` | Failed items |
| `pending` | Pending items |

### `result`

The run's headline result. The per-scenario, per-provider detail is streamed to session-api as the run executes.

| Field | Description |
|-------|-------------|
| `summary` | Aggregated result metrics (string map) |

An evaluation run's `summary` holds these keys. Every value is a string.

| Key | Description |
|-----|-------------|
| `passRate` | Percentage of items that passed, to one decimal place (`"100.0"`) |
| `totalItems` | Items executed |
| `passedItems` | Items that passed |
| `failedItems` | Items that failed |
| `avgDurationMs` | Average item duration in milliseconds |
| `totalTokens` | Tokens used; set only when above zero |
| `totalCost` | Cost to four decimal places; set only when above zero |
| `details` | JSON string with the breakdown: `scenarios`, `providers`, `cells` (one per scenario and provider pair), `assertions` and `errors` |

### `conditions`

| Type | Description |
|------|-------------|
| `Ready` | Overall readiness of the run |
| `SourceValid` | Referenced ArenaProject exists and is `Ready`, and the run's commit is pinned in `status.source` |
| `JobCreated` | Worker Kubernetes Job has been created |
| `Progressing` | Run is executing workers |
| `QueueRedisDrifted` | `True` (reason `ServiceGroupRedisChanged`) when the run's service group `redis` has changed since the run was pinned. The run keeps its queue. Set back to `False` (reason `Pinned`) if the group returns to the pinned Redis |
| `WithinWorkspaceQuota` | Present while the workspace sets [`spec.quotas.arena`](/reference/core/workspace/#quotasarena), written before the run starts. `True` (reason `QuotaSatisfied`) when the run is admitted. `False` while it waits, `Pending` with `Ready=False` (reason `WaitingForWorkspaceQuota`): reason `WaitingForConcurrentRunSlot` when `maxConcurrentJobs` runs are running, `WaitingForDailyRunQuota` when `maxJobsPerDay` runs were admitted in the last 24 hours (the message gives when the next slot opens), `WaitingForQuotaAdmission` until the quota has counted the run, `AdmissionReleased` when the run was admitted but did not start its workers within 15 minutes and gave its slot back. Once admitted, a `Ready=False WaitingForWorkspaceQuota` is removed. `False` with reason `WorkersClamped` when the run is admitted but asks for more workers than `maxWorkersPerJob`: it runs at the cap. A waiting run is never failed, and is admitted oldest first |
| `OutputPersisted` | Set once the run finishes, only for S3 output. `True` (reason `OutputUploaded`) when every worker uploaded; `False` (reason `OutputUploadFailed`, with a matching Warning event) when any worker's upload failed. It does not change the run's phase. |

### Other fields

| Field | Description |
|-------|-------------|
| `startTime` | When the run started |
| `completionTime` | When the run completed |
| `activeWorkers` | Current number of active worker pods |
| `workers` | The worker replicas the worker Job was created with: `template.workers.replicas`, capped at the workspace's `maxWorkersPerJob`. Set once, so a quota changed later never resizes a running run. A run whose Job predates the field gets the Job's `parallelism` |
| `observedGeneration` | Most recent generation observed by the controller |

## Complete Examples

### Basic Evaluation Job

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
metadata:
  name: basic-eval
  namespace: arena
spec:
  sourceRef:
    name: my-source
  arenaFile: config.arena.yaml
```

### Multi-Worker Evaluation

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
metadata:
  name: parallel-eval
  namespace: arena
spec:
  sourceRef:
    name: provider-comparison
  type: evaluation
  workers:
    replicas: 10
  output:
    type: s3
    s3:
      bucket: arena-results
      prefix: "evals/parallel/"
```

### Scheduled Nightly Evaluation

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
metadata:
  name: nightly-eval
  namespace: arena
spec:
  sourceRef:
    name: production-tests
  type: evaluation
  workers:
    replicas: 5
  output:
    type: s3
    s3:
      bucket: arena-results
      prefix: "evals/nightly/"
  schedule:
    cron: "0 2 * * *"
    timezone: "UTC"
  ttlSecondsAfterFinished: 604800  # 7 days
```

### Load Testing Job

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
metadata:
  name: provider-loadtest
  namespace: arena
spec:
  sourceRef:
    name: load-test-source
  type: loadtest
  trials: 5000
  sessionRecording: false
  loadTest:
    concurrency: 100
    vusPerWorker: 20
    ramp:
      up: 2m
      down: 1m
    thresholds:
      - metric: latency_p95
        operator: "<"
        value: "3s"
      - metric: error_rate
        operator: "<="
        value: "0.02"
  workers:
    replicas: 5
  output:
    type: s3
    s3:
      bucket: loadtest-results
      prefix: "loadtests/"
```

### Data Generation Job

```yaml
apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ArenaJob
metadata:
  name: synthetic-data
  namespace: arena
spec:
  sourceRef:
    name: datagen-source
  type: datagen
  dataGen:
    count: 10000
    format: jsonl
  workers:
    replicas: 4
  output:
    type: pvc
    pvc:
      claimName: generated-data
      subPath: "batch-001/"
```

## Workflow

1. **Create an ArenaProject** — a bundle containing the arena config file (`config.arena.yaml`) plus scenarios and prompts.
2. **Create an ArenaJob** — reference the project, select the config file with `arenaFile`, and specify execution parameters (job type, providers, workers).
3. **Monitor progress** — the job creates an ArenaRun per execution (`kubectl get arun`); watch the run's `status.progress` for completion.
4. **Retrieve results** — access results from the configured output destination.

## Related Resources

- **[ArenaProject](/reference/evaluation/arenaproject/)**: Defines the bundle source (git, OCI, S3, ConfigMap, or workspace)
- **[Arena Config File](/reference/evaluation/arenaconfig/)**: Schema of the `config.arena.yaml` file inside the bundle
- **[Provider](/reference/core/provider/)**: LLM provider configuration