Monitor Arena jobs

This guide covers how to monitor Arena Fleet jobs in real-time, track progress, and access evaluation results.

Jobs and runs

An ArenaJob is a template: what to evaluate and, optionally, the cron schedule to do it on. Each execution is a separate ArenaRun, which is where phase, progress and results live. That separation is what lets a scheduled job keep the history of what it has done — a single object would overwrite last night’s result on every tick.

So monitoring is two steps: find the run, then watch it.

# Every run of a job, newest last
kubectl get arenaruns -l omnia.altairalabs.ai/arena-job=my-eval

# Or jump straight to the most recent
kubectl get arenajob my-eval -o jsonpath='{.status.lastRunRef.name}'

An ArenaJob’s own status describes scheduling rather than execution:

status:
  active:
    - name: my-eval-x7k2p     # runs currently executing
  lastRunRef:
    name: my-eval-x7k2p
  lastScheduleTime: "2026-01-18T02:00:00Z"
  nextScheduleTime: "2026-01-19T02:00:00Z"

Checking run status

Basic status

View the current state of a run:

kubectl get arenarun my-eval-x7k2p

Output:

NAME            JOB       TRIGGER    PHASE     PROGRESS   AGE
my-eval-x7k2p   my-eval   Schedule   Running   45         2m

Detailed status

Get full status details:

kubectl get arenarun my-eval-x7k2p -o yaml

Key status fields:

status:
  phase: Running
  progress:
    total: 100      # Total scenarios to evaluate
    completed: 45   # Successfully completed
    failed: 2       # Failed evaluations
    pending: 53     # Waiting to run
  activeWorkers: 3
  startTime: "2025-01-18T10:00:00Z"
  conditions:
    - type: Ready
      status: "True"
    - type: Progressing
      status: "True"
      message: "45/100 scenarios completed"

Watch progress in real-time

Monitor job progress as it runs:

kubectl get arenarun my-eval-x7k2p -w

Or use watch for periodic updates:

watch -n 5 kubectl get arenarun my-eval-x7k2p

Understanding run phases

Phase	Description
Pending	Job created, waiting to start
Running	Workers are actively processing scenarios
Succeeded	All scenarios completed successfully
Failed	Job failed (threshold exceeded or error)
Cancelled	Job was manually cancelled

Viewing worker status

List worker pods

kubectl get pods -l arena.omnia.altairalabs.ai/job=my-eval

Output:

NAME                    READY   STATUS    RESTARTS   AGE
my-eval-worker-abc12    1/1     Running   0          2m
my-eval-worker-def34    1/1     Running   0          2m
my-eval-worker-ghi56    1/1     Running   0          2m

View worker logs

Stream logs from all workers:

kubectl logs -l arena.omnia.altairalabs.ai/job=my-eval -f

Logs from a specific worker:

kubectl logs my-eval-worker-abc12 -f

Check worker resource usage

kubectl top pods -l arena.omnia.altairalabs.ai/job=my-eval

Accessing results

From job status

For completed jobs, results are summarized in the status:

kubectl get arenarun my-eval-x7k2p -o jsonpath='{.status.result.summary}'

Result URL

If output storage is configured, get the result location:

kubectl get arenarun my-eval-x7k2p -o jsonpath='{.status.result.url}'

Download results

For S3 storage:

# Get the result prefix
RESULT_URL=$(kubectl get arenarun my-eval-x7k2p -o jsonpath='{.status.result.url}')
aws s3 cp $RESULT_URL/results.json ./results.json

For PVC storage:

# Port-forward or exec into a pod to access PVC
kubectl cp <pod>:/path/to/results ./results

Prometheus metrics

Arena Fleet exposes metrics for monitoring with Prometheus.

Key metrics

Metric	Description
arena_job_phase	Current job phase (gauge)
arena_job_progress_total	Total scenarios in job
arena_job_progress_completed	Completed scenarios
arena_job_progress_failed	Failed scenarios
arena_job_duration_seconds	Job execution duration
arena_scenario_latency_seconds	Per-scenario LLM latency
arena_scenario_tokens_total	Token usage per scenario

Example Prometheus queries

Total running jobs:

count(arena_job_phase{phase="Running"})

Job completion rate:

arena_job_progress_completed / arena_job_progress_total

Average scenario latency:

avg(arena_scenario_latency_seconds) by (job_name, provider)

Failed scenario rate:

rate(arena_job_progress_failed[5m])

Grafana dashboard

If Grafana is enabled, Arena metrics are available for visualization.

Sample dashboard panels

Job Progress:

arena_job_progress_completed{job_name="$job"}

Scenario Latency Histogram:

histogram_quantile(0.95, arena_scenario_latency_seconds_bucket)

Token Usage Over Time:

sum(rate(arena_scenario_tokens_total[5m])) by (provider)

Event monitoring

View events related to Arena jobs:

kubectl get events --field-selector involvedObject.name=my-eval

Key events to watch for:

Event	Meaning
JobStarted	Job execution began
WorkersCreated	Worker pods created
ScenarioCompleted	Individual scenario finished
JobSucceeded	Job completed successfully
JobFailed	Job failed
RetryScheduled	Failed scenario being retried

Setting up alerts

Alert on job failure

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: arena-alerts
spec:
  groups:
    - name: arena
      rules:
        - alert: ArenaJobFailed
          expr: arena_job_phase{phase="Failed"} == 1
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: "Arena job {{ $labels.job_name }} failed"
            description: "Job has been in Failed state for more than 1 minute"

Alert on high failure rate

- alert: ArenaHighFailureRate
  expr: |
    (arena_job_progress_failed / arena_job_progress_total) > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Arena job {{ $labels.job_name }} has >10% failure rate"

Alert on slow evaluations

- alert: ArenaSlowEvaluation
  expr: |
    avg(arena_scenario_latency_seconds) by (job_name) > 60
  for: 10m
  labels:
    severity: info
  annotations:
    summary: "Arena job {{ $labels.job_name }} has slow evaluations (>60s avg)"

Cancelling

Cancellation applies to an execution, so it is set on the run:

kubectl patch arenarun my-eval-x7k2p --type=merge -p '{"spec":{"cancelled":true}}'

The operator deletes the worker Job and moves the run to Cancelled. This has no effect once a run has already finished.

Deleting the job removes the template and, with it, every run it owns:

kubectl delete arenajob my-eval

To stop future executions without losing the runs already recorded, remove the schedule instead:

kubectl patch arenajob my-eval --type=json -p '[{"op":"remove","path":"/spec/schedule"}]'

Debugging failed jobs

Check job conditions

kubectl describe arenajob my-eval | grep -A 10 Conditions

View failed scenarios

Check worker logs for failures:

kubectl logs -l arena.omnia.altairalabs.ai/job=my-eval | grep -i error

Common failure reasons

Reason	Resolution
ConfigNotReady	Verify the source bundle contains a valid config.arena.yaml
SourceFetchFailed	Verify ArenaSource can fetch bundle
ProviderError	Check provider credentials and limits
Timeout	Increase evaluation timeout
AssertionFailed	Expected behavior - check test assertions

Related resources

• Troubleshoot Arena: Debug common issues
• ArenaJob Reference: Complete status field documentation
• Set Up Observability: Configure Prometheus and Grafana