Troubleshoot Arena Fleet

Enterprise

Find the failing Arena resource and fix its cause.

Prerequisites

Arena installed (enterprise.enabled: true) and kubectl access to the project’s namespace.

Find the failing resource

An ArenaJob is a template. Each execution is an ArenaRun that the job creates, and the run holds the phase, progress and result. See ArenaJob and ArenaProject for every field.

Check the project. A run cannot start until the project is Ready.
Terminal window
```
kubectl get arenaproject
kubectl describe arenaproject my-project
```

Find the latest run of the job and read its conditions.

kubectl get arenajob my-job -o jsonpath='{.status.lastRunRef.name}'
kubectl describe arenarun <run-name>

Read the worker logs. The worker pods carry the label omnia.altairalabs.ai/job=<run-name>.
Terminal window
```
kubectl logs -l omnia.altairalabs.ai/job=<run-name>
```
If no worker pod exists, read the Arena controller logs.
Terminal window
```
kubectl logs -n omnia-system deployment/omnia-arena-controller
```
The Arena controllers run in the <release>-arena-controller Deployment. The examples assume the release is omnia.

Verify

The project shows Ready, and the run moves from Pending to Running.

Troubleshooting

A project stays in Pending

A hosted project is Pending while its seed waits for its ConfigMap. Read the Seeded condition:

kubectl get arenaproject my-project \
  -o jsonpath='{.status.conditions[?(@.type=="Seeded")]}'

Create the ConfigMap named by spec.repository.hosted.seed.configMap.name in the project’s namespace. The seed is applied once, as the repository’s first commit. A repository that already has commits never waits for it.

An external project is in Error

The project shows phase Error with Ready=False, reason RepositoryFailed. The message names the fetch failure.

kubectl describe arenaproject my-project

A wrong URL, or a private repository with no credential. spec.repository.external.url must be an https:// URL. Set readSecretRef to a Secret holding username and password, or a single token.
The message says the read secret is not labelled. Only a Secret labelled omnia.altairalabs.ai/credential-for=git is read.
Terminal window
```
kubectl -n <namespace> label secret git-credentials omnia.altairalabs.ai/credential-for=git
```
DNS or egress from the Arena controller to the git host is blocked.

A failed fetch is retried, and a healthy external project is fetched every pollInterval (default 5m).

A run fails with RefNotFound, or an edit is missing from a run

A run executes a pushed commit. The job’s spec.ref names nothing in the repository when the branch or tag does not exist or the commit was never pushed. Uncommitted and unpushed edits are visible only to you. Commit, then select Sync on the project editor’s Source control tab, and run again.

The builder cannot start a run: “commit and push first”

The builder’s arena_start_run runs the commit your working clone has checked out, once it is pushed. The 409 response names the uncommitted changes, or how many commits the branch is ahead. The builder commits but does not push. Select Sync on the Source control tab, then ask the builder to start the run again.

“Your workspace was left idle with unsaved work”

Your working clone went unused for the idle period (14 days by default) while it held work the repository lacks. The Arena controller keeps and flags such a clone. Commit and Sync (or push) the work, or discard it. Any change you make in the clone clears the flag. To list a project’s flagged clones:

kubectl get arenaproject <project> -n <workspace-namespace> \
  -o jsonpath='{range .status.workspaces[?(@.state=="Flagged")]}{.lastActivity}{"\t"}{.reason}{"\n"}{end}'

My workspace is back at the last pushed commit

Your clone went unused for the idle period with nothing unpushed, so the Arena controller deleted it and opening the project cloned it again. To keep clones longer, raise enterprise.arena.controller.workspaceIdleAfter in the chart values.

The run is Pending with SourceNotReady

The run’s Ready condition has reason SourceNotReady, and the message reads arenaProject my-project is not ready (phase "Pending") or arenaProject my-project not found. The run proceeds when the project is Ready. Fix the project using the two sections above.

The run is Pending with no worker pod

Describe the Job named <run-name>-worker:

kubectl describe job <run-name>-worker
kubectl get events --field-selector involvedObject.name=<run-name>-worker

Not enough capacity for the pods. Lower spec.workers.replicas or add nodes.
The worker image cannot be pulled. Check the image pull secrets and the enterprise.arena.worker.image values.

Job creation fails (JobCreated is False)

The run’s JobCreated condition has reason JobCreationFailed. Common messages:

provider group validation failed: arena config requires provider groups [...] but spec.providers is missing: [...]. Map every provider group the arena config names, including default, in spec.providers. Each entry is a providerRef or an agentRef.
work item matrix exceeds the limit. The scenarios, providers and trials multiply to more items than the job type allows. The run fails with reason WorkItemCapExceeded. Reduce spec.trials, the scenarios or the providers.

The worker cannot find the config file

The worker log shows arena config file not found in bundle, or a parse or load error. The worker reads spec.arenaFile (default config.arena.yaml). If that file is missing, it looks for config.arena.yaml, config.arena.yml, arena.yaml, arena.yml, config.yaml and config.yml in the bundle root. spec.arenaFile names one file and rejects glob characters.

Check that the pushed project contains the file and that its YAML parses.

The run executes more scenarios than the filter allows

When spec.scenarios.include and exclude match no scenario, the controller cannot build a scenario matrix. It queues one default item per provider, and the worker runs every scenario in the bundle for each.

kubectl get arenajob my-job -o jsonpath='{.spec.scenarios}'

Patterns match a scenario’s ID, its path, or its file name. Correct the patterns, or remove spec.scenarios to run every scenario.

Workers crash or restart

Worker pods show CrashLoopBackOff or repeated restarts.

kubectl logs -l omnia.altairalabs.ai/job=<run-name> --previous
kubectl describe pod <worker-pod-name>

OOMKilled. Raise the worker memory limit in the chart values (defaults: limit 512Mi, request 128Mi). The items the dead worker was running are requeued for the other workers as soon as the Job records the pod’s failure, and the run shows a WorkReclaimed event. An item that kills its worker on every attempt is written off as failed after its last attempt, so the run still finishes.
```
enterprise:
  arena:
    worker:
      resources:
        limits:
          memory: 1Gi
        requests:
          memory: 512Mi
```
Provider errors such as rate limits. Check the provider’s quota.
An invalid project. Fix the file that the log names.

Set spec.verbose: true on the job for debug logging in the worker.

kubectl get arenarun <run-name> -o jsonpath='{.status.progress}'
kubectl logs -l omnia.altairalabs.ai/job=<run-name> | grep -i "error\|failed"

Assertion failures are the evaluation result. Provider errors such as 429 Too Many Requests or context deadline exceeded are not: lower spec.workers.replicas, and for a load test spec.loadTest.concurrency.

Results are not stored

kubectl get arenarun <run-name> -o jsonpath='{.status.result}'
kubectl logs -l omnia.altairalabs.ai/job=<run-name> | grep -i "s3\|storage\|upload"

For S3 output, the Secret named by spec.output.s3.secretRef must hold the keys AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and its credentials must be able to write to the bucket. See Configure Arena S3 storage.

Workers do not process items

The run is Running and status.progress does not advance. The queue is in Redis.

Look for failed to connect to Redis in the Arena controller logs and in the worker logs.
Check enterprise.arena.queue.type and enterprise.arena.queue.redis in the chart values. Empty redis fields fall back to the chart’s redis.default.

Nothing reconciles

An ArenaProject or ArenaJob is created and nothing happens.

kubectl get deployment -n omnia-system omnia-arena-controller
kubectl get crd arenaprojects.omnia.altairalabs.ai arenajobs.omnia.altairalabs.ai arenaruns.omnia.altairalabs.ai

The Deployment exists only when enterprise.enabled is true. The CRDs come from the omnia-crds chart.