# Monitor Arena jobs Follow an Arena run from creation to results, and alert when it fails. ## Prerequisites - An [ArenaJob](/reference/evaluation/arenajob/) that has created at least one ArenaRun. - `kubectl` access to the namespace the job runs in. - For the metrics and alert steps, a Prometheus that scrapes pod annotations. ## Find the run An ArenaJob is a template. Each execution is an ArenaRun, and the run holds the phase, progress and results. The job's own status holds scheduling data only (`active`, `lastRunRef`, `lastScheduleTime`, `nextScheduleTime`). 1. List the runs of a job: ```bash kubectl get arenaruns -l omnia.altairalabs.ai/arena-job=my-eval ``` 2. Or print the most recent run's name: ```bash kubectl get arenajob my-eval -o jsonpath='{.status.lastRunRef.name}' ``` ## Follow the run 1. Check the phase: ```bash kubectl get arenarun my-eval-x7k2p ``` The columns are `READY`, `JOB`, `TRIGGER`, `PHASE` and `AGE`. Add `-o wide` for the `PROGRESS` column. See [phase](/reference/evaluation/arenajob/#phase) for the values. 2. Watch it change: ```bash kubectl get arenarun my-eval-x7k2p -w ``` 3. Read the status fields: ```bash kubectl get arenarun my-eval-x7k2p -o yaml ``` While the run is `Running`: - `status.progress.total` and `pending` hold the work-item count from creation, and `completed` and `failed` are `0`. `completed`, `failed` and `pending` are rewritten when the run reaches a terminal phase. - A load test with a `duration` generates its own work and has no `progress` until it ends. - `status.activeWorkers` is the number of worker pods running. - The conditions `SourceValid`, `JobCreated` and `Progressing` are `True`. `Ready` is set only when the run finishes. ## Check the workers The worker Job is `-worker`. Its pods carry the label `omnia.altairalabs.ai/job=`. 1. List the pods: ```bash kubectl get pods -l omnia.altairalabs.ai/job=my-eval-x7k2p ``` 2. Stream the logs of all workers: ```bash kubectl logs -l omnia.altairalabs.ai/job=my-eval-x7k2p -f ``` 3. Check resource use: ```bash kubectl top pods -l omnia.altairalabs.ai/job=my-eval-x7k2p ``` ## Read the results 1. Print the run's summary once it has finished: ```bash kubectl get arenarun my-eval-x7k2p -o jsonpath='{.status.result.summary}' ``` See [result](/reference/evaluation/arenajob/#result) for the keys. 2. Find where the output files went. The run does not record the destination, but it holds a frozen copy of the job's `output`: ```bash kubectl get arenarun my-eval-x7k2p -o jsonpath='{.spec.template.output}' ``` 3. For S3 output, list the bucket. The worker uploads after all items finish, under `//`: ```bash aws s3 ls s3:////my-eval/ --recursive ``` 4. For PVC output, mount the claim named in `output.pvc.claimName` (and its `subPath`) in a pod, then copy the files out: ```bash kubectl cp :/path/to/results ./results ``` ## Watch metrics Each worker pod carries the annotations `prometheus.io/scrape: "true"`, port `9090` and path `/metrics`. Worker metrics use the label `job_name`, whose value is the ArenaRun name. The [Arena worker](/reference/platform/observability-metrics/#arena-worker) section of the platform metrics reference lists every metric and label. Failed work items per run over five minutes: ```promql sum by (job_name) (increase(omnia_arena_work_items_total{status="fail"}[5m])) ``` Token rate by provider: ```promql sum by (provider) (rate(omnia_arena_tokens_total[5m])) ``` When `observability.enabled` and `grafana.enabled` are both `true`, the chart provisions an "Omnia Arena Job" dashboard (uid `omnia-arena-job`) with a `job_name` selector. See [Set up observability](/how-to/observability/setup-observability/). ## Read events Run events are recorded against the run, and job events against the job: ```bash kubectl get events --field-selector involvedObject.name=my-eval-x7k2p kubectl get events --field-selector involvedObject.name=my-eval ``` | Object | Events | |--------|--------| | Run | `JobCreated`, `JobSucceeded`, `JobFailed`, `BudgetExceeded`, `ConfigNotReady`, `UnlicensedCapability` (a job type or schedule the licence does not cover; the run proceeds), `LicenseViolation` (worker replicas over the licence's limit), and the refusal reasons `RefNotFound`, `SnapshotUnexportable`, `WorkItemCapExceeded`, `ServiceGroupAmbiguous`, `InvalidProviderEntry`, and `QueueStateLost` (the run's work queue vanished from Redis mid-run) | | Job | `RunCreated`, `RunSkipped`, `RunReplaced`, `Suspended`, `InvalidSchedule` | ## Alert on failures Apply a PrometheusRule. The label values are illustrative. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: arena-alerts spec: groups: - name: arena rules: - alert: ArenaItemsFailing expr: | sum by (job_name) (increase(omnia_arena_work_items_total{status="fail"}[10m])) > 0 for: 1m labels: severity: warning annotations: summary: "Arena run {{ $labels.job_name }} has failing work items" - alert: ArenaSlowItems expr: | histogram_quantile(0.95, sum by (le, job_name) (rate(omnia_arena_work_item_duration_seconds_bucket[5m]))) > 60 for: 10m labels: severity: info annotations: summary: "Arena run {{ $labels.job_name }} p95 item duration is over 60s" ``` ## Cancel or stop 1. Cancel one run. The operator deletes the worker Job, drains the run's queued work and moves the run to `Cancelled`. It has no effect on a run that has finished: ```bash kubectl patch arenarun my-eval-x7k2p --type=merge -p '{"spec":{"cancelled":true}}' ``` 2. Stop future runs and keep the recorded ones by removing the schedule: ```bash kubectl patch arenajob my-eval --type=json -p '[{"op":"remove","path":"/spec/schedule"}]' ``` 3. Delete the job and every run it owns: ```bash kubectl delete arenajob my-eval ``` ## Verify The run reaches `Succeeded`, and `kubectl get arenarun my-eval-x7k2p -o jsonpath='{.status.result.summary.passRate}'` prints a pass rate. ## Troubleshooting ### The run stays Pending The ArenaProject is missing or not `Ready`. The run's `SourceValid` condition is `False` with reason `SourceNotReady`, and a `ConfigNotReady` warning event names the cause. The controller retries on its own. ```bash kubectl describe arenarun my-eval-x7k2p ``` ### The run is Failed The `Progressing` condition carries the reason (`Ready` carries `LicenseViolation`): | Reason | Meaning | |--------|---------| | `TestsFailed` | Some items failed | | `NoTestsRan` | The run finished but no item produced a result | | `BudgetExceeded` | The run hit its `budgetLimit` | | `JobFailed` | The worker Job failed; the message says why | | `QueueStateLost` | The run's work queue vanished from Redis while it ran: Redis restarted without persistence, was flushed, or evicted the run's keys. The remaining items, progress and results cannot be recovered, so the run fails at once and its workers are stopped. Run the job again, and give the queue's Redis persistence and `noeviction` | | `RefNotFound`, `SnapshotUnexportable`, `WorkItemCapExceeded`, `ServiceGroupAmbiguous`, `InvalidProviderEntry` | The run was refused; the message says what to change | | `LicenseViolation` | The run asked for more worker replicas than the licence's `maxWorkerReplicas`. Lower `workers.replicas` or install a licence with a higher limit | A work item that runs longer than 10 minutes fails with `work item timed out after 10m0s`. Provider errors and failed assertions appear in the worker logs. ### The run succeeded but the S3 bucket is empty The upload runs after the work finishes. A failure is logged by the worker and does not fail the run: ```bash kubectl logs -l omnia.altairalabs.ai/job=my-eval-x7k2p | grep -i "s3" ``` ## Related pages - [Troubleshoot Arena](/how-to/evaluation/troubleshoot-arena/) - [ArenaJob and ArenaRun reference](/reference/evaluation/arenajob/) - [Set up observability](/how-to/observability/setup-observability/)