Monitor Arena jobs

Follow an Arena run from creation to results, and alert when it fails.

Prerequisites

• An ArenaJob that has created at least one ArenaRun.
• kubectl access to the namespace the job runs in.
• For the metrics and alert steps, a Prometheus that scrapes pod annotations.

Find the run

An ArenaJob is a template. Each execution is an ArenaRun, and the run holds the phase, progress and results. The job’s own status holds scheduling data only (active, lastRunRef, lastScheduleTime, nextScheduleTime).

1. List the runs of a job:

   kubectl get arenaruns -l omnia.altairalabs.ai/arena-job=my-eval

2. Or print the most recent run’s name:

   kubectl get arenajob my-eval -o jsonpath='{.status.lastRunRef.name}'

Follow the run

1. Check the phase:

   kubectl get arenarun my-eval-x7k2p

   The columns are READY, JOB, TRIGGER, PHASE and AGE. Add -o wide for the PROGRESS column. See phase for the values.

2. Watch it change:

   kubectl get arenarun my-eval-x7k2p -w

3. Read the status fields:

   kubectl get arenarun my-eval-x7k2p -o yaml

   While the run is Running:

   • status.progress.total and pending hold the work-item count from creation, and completed and failed are 0. completed, failed and pending are rewritten when the run reaches a terminal phase.
   • A load test with a duration generates its own work and has no progress until it ends.
   • status.activeWorkers is the number of worker pods running.
   • The conditions SourceValid, JobCreated and Progressing are True. Ready is set only when the run finishes.

Check the workers

The worker Job is <run-name>-worker. Its pods carry the label omnia.altairalabs.ai/job=<run-name>.

1. List the pods:

   kubectl get pods -l omnia.altairalabs.ai/job=my-eval-x7k2p

2. Stream the logs of all workers:

   kubectl logs -l omnia.altairalabs.ai/job=my-eval-x7k2p -f

3. Check resource use:

   kubectl top pods -l omnia.altairalabs.ai/job=my-eval-x7k2p

Read the results

1. Print the run’s summary once it has finished:

   kubectl get arenarun my-eval-x7k2p -o jsonpath='{.status.result.summary}'

   See result for the keys.

2. Find where the output files went. The run does not record the destination, but it holds a frozen copy of the job’s output:

   kubectl get arenarun my-eval-x7k2p -o jsonpath='{.spec.template.output}'

3. For S3 output, list the bucket. The worker uploads after all items finish, under <prefix>/<job-name>/:

   aws s3 ls s3://<bucket>/<prefix>/my-eval/ --recursive

4. For PVC output, mount the claim named in output.pvc.claimName (and its subPath) in a pod, then copy the files out:

   kubectl cp <pod>:/path/to/results ./results

Watch metrics

Each worker pod carries the annotations prometheus.io/scrape: "true", port 9090 and path /metrics. Worker metrics use the label job_name, whose value is the ArenaRun name.

The Arena worker section of the platform metrics reference lists every metric and label.

Failed work items per run over five minutes:

sum by (job_name) (increase(omnia_arena_work_items_total{status="fail"}[5m]))

Token rate by provider:

sum by (provider) (rate(omnia_arena_tokens_total[5m]))

When observability.enabled and grafana.enabled are both true, the chart provisions an “Omnia Arena Job” dashboard (uid omnia-arena-job) with a job_name selector. See Set up observability.

Read events

Run events are recorded against the run, and job events against the job:

kubectl get events --field-selector involvedObject.name=my-eval-x7k2p
kubectl get events --field-selector involvedObject.name=my-eval

Object	Events
Run	JobCreated, JobSucceeded, JobFailed, BudgetExceeded, ConfigNotReady, UnlicensedCapability (a job type or schedule the licence does not cover; the run proceeds), LicenseViolation (worker replicas over the licence’s limit), and the refusal reasons RefNotFound, SnapshotUnexportable, WorkItemCapExceeded, ServiceGroupAmbiguous, InvalidProviderEntry, and QueueStateLost (the run’s work queue vanished from Redis mid-run)
Job	RunCreated, RunSkipped, RunReplaced, Suspended, InvalidSchedule

Alert on failures

Apply a PrometheusRule. The label values are illustrative.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: arena-alerts
spec:
  groups:
    - name: arena
      rules:
        - alert: ArenaItemsFailing
          expr: |
            sum by (job_name) (increase(omnia_arena_work_items_total{status="fail"}[10m])) > 0
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: "Arena run {{ $labels.job_name }} has failing work items"
        - alert: ArenaSlowItems
          expr: |
            histogram_quantile(0.95, sum by (le, job_name) (rate(omnia_arena_work_item_duration_seconds_bucket[5m]))) > 60
          for: 10m
          labels:
            severity: info
          annotations:
            summary: "Arena run {{ $labels.job_name }} p95 item duration is over 60s"

Cancel or stop

1. Cancel one run. The operator deletes the worker Job, drains the run’s queued work and moves the run to Cancelled. It has no effect on a run that has finished:

   kubectl patch arenarun my-eval-x7k2p --type=merge -p '{"spec":{"cancelled":true}}'

2. Stop future runs and keep the recorded ones by removing the schedule:

   kubectl patch arenajob my-eval --type=json -p '[{"op":"remove","path":"/spec/schedule"}]'

3. Delete the job and every run it owns:

   kubectl delete arenajob my-eval

Verify

The run reaches Succeeded, and kubectl get arenarun my-eval-x7k2p -o jsonpath='{.status.result.summary.passRate}' prints a pass rate.

Troubleshooting

The run stays Pending

The ArenaProject is missing or not Ready. The run’s SourceValid condition is False with reason SourceNotReady, and a ConfigNotReady warning event names the cause. The controller retries on its own.

kubectl describe arenarun my-eval-x7k2p

The run is Failed

The Progressing condition carries the reason (Ready carries LicenseViolation):

Reason	Meaning
TestsFailed	Some items failed
NoTestsRan	The run finished but no item produced a result
BudgetExceeded	The run hit its budgetLimit
JobFailed	The worker Job failed; the message says why
QueueStateLost	The run’s work queue vanished from Redis while it ran: Redis restarted without persistence, was flushed, or evicted the run’s keys. The remaining items, progress and results cannot be recovered, so the run fails at once and its workers are stopped. Run the job again, and give the queue’s Redis persistence and noeviction
RefNotFound, SnapshotUnexportable, WorkItemCapExceeded, ServiceGroupAmbiguous, InvalidProviderEntry	The run was refused; the message says what to change
LicenseViolation	The run asked for more worker replicas than the licence’s maxWorkerReplicas. Lower workers.replicas or install a licence with a higher limit

A work item that runs longer than 10 minutes fails with work item timed out after 10m0s. Provider errors and failed assertions appear in the worker logs.

The run succeeded but the S3 bucket is empty

The upload runs after the work finishes. A failure is logged by the worker and does not fail the run:

kubectl logs -l omnia.altairalabs.ai/job=my-eval-x7k2p | grep -i "s3"

Related pages

• Troubleshoot Arena
• ArenaJob and ArenaRun reference
• Set up observability