# Troubleshoot Arena Fleet Find the failing Arena resource and fix its cause. ## Prerequisites - Arena installed (`enterprise.enabled: true`) and `kubectl` access to the project's namespace. ## Find the failing resource An ArenaJob is a template. Each execution is an ArenaRun that the job creates, and the run holds the phase, progress and result. See [ArenaJob](/reference/evaluation/arenajob/) and [ArenaProject](/reference/evaluation/arenaproject/) for every field. 1. Check the project. A run cannot start until the project is `Ready`. ```bash kubectl get arenaproject kubectl describe arenaproject my-project ``` 2. Find the latest run of the job and read its conditions. ```bash kubectl get arenajob my-job -o jsonpath='{.status.lastRunRef.name}' kubectl describe arenarun ``` 3. Read the worker logs. The worker pods carry the label `omnia.altairalabs.ai/job=`. ```bash kubectl logs -l omnia.altairalabs.ai/job= ``` 4. If no worker pod exists, read the Arena controller logs. ```bash kubectl logs -n omnia-system deployment/omnia-arena-controller ``` The Arena controllers run in the `-arena-controller` Deployment. The examples assume the release is `omnia`. ## Verify The project shows `Ready`, and the run moves from `Pending` to `Running`. ## Troubleshooting ### A project stays in Pending A hosted project is `Pending` while its seed waits for its ConfigMap. Read the `Seeded` condition: ```bash kubectl get arenaproject my-project \ -o jsonpath='{.status.conditions[?(@.type=="Seeded")]}' ``` Create the ConfigMap named by `spec.repository.hosted.seed.configMap.name` in the project's namespace. The seed is applied once, as the repository's first commit. A repository that already has commits never waits for it. ### An external project is in Error The project shows phase `Error` with `Ready=False`, reason `RepositoryFailed`. The message names the fetch failure. ```bash kubectl describe arenaproject my-project ``` - A wrong URL, or a private repository with no credential. `spec.repository.external.url` must be an `https://` URL. Set `readSecretRef` to a Secret holding `username` and `password`, or a single `token`. - The message says the read secret is not labelled. Only a Secret labelled `omnia.altairalabs.ai/credential-for=git` is read. ```bash kubectl -n label secret git-credentials omnia.altairalabs.ai/credential-for=git ``` - DNS or egress from the Arena controller to the git host is blocked. A failed fetch is retried, and a healthy external project is fetched every `pollInterval` (default `5m`). ### A run fails with RefNotFound, or an edit is missing from a run A run executes a pushed commit. The job's `spec.ref` names nothing in the repository when the branch or tag does not exist or the commit was never pushed. Uncommitted and unpushed edits are visible only to you. Commit, then select **Sync** on the project editor's Source control tab, and run again. ### The builder cannot start a run: "commit and push first" The builder's `arena_start_run` runs the commit your working clone has checked out, once it is pushed. The `409` response names the uncommitted changes, or how many commits the branch is ahead. The builder commits but does not push. Select **Sync** on the Source control tab, then ask the builder to start the run again. ### "Your workspace was left idle with unsaved work" Your working clone went unused for the idle period (14 days by default) while it held work the repository lacks. The Arena controller keeps and flags such a clone. Commit and **Sync** (or push) the work, or discard it. Any change you make in the clone clears the flag. To list a project's flagged clones: ```bash kubectl get arenaproject -n \ -o jsonpath='{range .status.workspaces[?(@.state=="Flagged")]}{.lastActivity}{"\t"}{.reason}{"\n"}{end}' ``` ### My workspace is back at the last pushed commit Your clone went unused for the idle period with nothing unpushed, so the Arena controller deleted it and opening the project cloned it again. To keep clones longer, raise `enterprise.arena.controller.workspaceIdleAfter` in the chart values. ### The run is Pending with SourceNotReady The run's `Ready` condition has reason `SourceNotReady`, and the message reads `arenaProject my-project is not ready (phase "Pending")` or `arenaProject my-project not found`. The run proceeds when the project is `Ready`. Fix the project using the two sections above. ### The run is Pending with no worker pod Describe the Job named `-worker`: ```bash kubectl describe job -worker kubectl get events --field-selector involvedObject.name=-worker ``` - Not enough capacity for the pods. Lower `spec.workers.replicas` or add nodes. - The worker image cannot be pulled. Check the image pull secrets and the `enterprise.arena.worker.image` values. ### Job creation fails (JobCreated is False) The run's `JobCreated` condition has reason `JobCreationFailed`. Common messages: - `provider group validation failed: arena config requires provider groups [...] but spec.providers is missing: [...]`. Map every provider group the arena config names, including `default`, in `spec.providers`. Each entry is a `providerRef` or an `agentRef`. - `work item matrix exceeds the limit`. The scenarios, providers and trials multiply to more items than the job type allows. The run fails with reason `WorkItemCapExceeded`. Reduce `spec.trials`, the scenarios or the providers. ### The worker cannot find the config file The worker log shows `arena config file not found in bundle`, or a parse or load error. The worker reads `spec.arenaFile` (default `config.arena.yaml`). If that file is missing, it looks for `config.arena.yaml`, `config.arena.yml`, `arena.yaml`, `arena.yml`, `config.yaml` and `config.yml` in the bundle root. `spec.arenaFile` names one file and rejects glob characters. Check that the pushed project contains the file and that its YAML parses. ### The run executes more scenarios than the filter allows When `spec.scenarios.include` and `exclude` match no scenario, the controller cannot build a scenario matrix. It queues one `default` item per provider, and the worker runs every scenario in the bundle for each. ```bash kubectl get arenajob my-job -o jsonpath='{.spec.scenarios}' ``` Patterns match a scenario's ID, its path, or its file name. Correct the patterns, or remove `spec.scenarios` to run every scenario. ### Workers crash or restart Worker pods show `CrashLoopBackOff` or repeated restarts. ```bash kubectl logs -l omnia.altairalabs.ai/job= --previous kubectl describe pod ``` - `OOMKilled`. Raise the worker memory limit in the chart values (defaults: limit `512Mi`, request `128Mi`). The items the dead worker was running are requeued for the other workers as soon as the Job records the pod's failure, and the run shows a `WorkReclaimed` event. An item that kills its worker on every attempt is written off as failed after its last attempt, so the run still finishes. ```yaml enterprise: arena: worker: resources: limits: memory: 1Gi requests: memory: 512Mi ``` - Provider errors such as rate limits. Check the provider's quota. - An invalid project. Fix the file that the log names. Set `spec.verbose: true` on the job for debug logging in the worker. ### A high share of scenarios fail ```bash kubectl get arenarun -o jsonpath='{.status.progress}' kubectl logs -l omnia.altairalabs.ai/job= | grep -i "error\|failed" ``` Assertion failures are the evaluation result. Provider errors such as `429 Too Many Requests` or `context deadline exceeded` are not: lower `spec.workers.replicas`, and for a load test `spec.loadTest.concurrency`. ### Results are not stored ```bash kubectl get arenarun -o jsonpath='{.status.result}' kubectl logs -l omnia.altairalabs.ai/job= | grep -i "s3\|storage\|upload" ``` For S3 output, the Secret named by `spec.output.s3.secretRef` must hold the keys `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, and its credentials must be able to write to the bucket. See [Configure Arena S3 storage](/how-to/evaluation/configure-arena-s3-storage/). ### Workers do not process items The run is `Running` and `status.progress` does not advance. The queue is in Redis. - Look for `failed to connect to Redis` in the Arena controller logs and in the worker logs. - Check `enterprise.arena.queue.type` and `enterprise.arena.queue.redis` in the chart values. Empty `redis` fields fall back to the chart's `redis.default`. ### Nothing reconciles An ArenaProject or ArenaJob is created and nothing happens. ```bash kubectl get deployment -n omnia-system omnia-arena-controller kubectl get crd arenaprojects.omnia.altairalabs.ai arenajobs.omnia.altairalabs.ai arenaruns.omnia.altairalabs.ai ``` The Deployment exists only when `enterprise.enabled` is `true`. The CRDs come from the `omnia-crds` chart. ## Related pages - [Arena Fleet architecture](/explanation/evaluation/arena-fleet/) - [ArenaProject](/reference/evaluation/arenaproject/), [Arena config file](/reference/evaluation/arenaconfig/) and [ArenaJob](/reference/evaluation/arenajob/) - [Monitor Arena jobs](/how-to/evaluation/monitor-arena-jobs/)