Configure pod overrides
Set podOverrides on an AgentRuntime, a Workspace service group or an ArenaJob to change the pods the operator generates for it.
Prerequisites
Section titled “Prerequisites”- The CRD that generates the pod, and permission to edit it.
- For a CSI secret store: the CSI driver and a
SecretProviderClassin the pod’s namespace. - For workload identity: a ServiceAccount that is federated with your cloud.
Choose the placement
Section titled “Choose the placement”podOverrides is accepted in these fields. podOverrides lists the pod each one affects and how every field merges.
| Goal | Field |
|---|---|
| Agent pod (facade and runtime) | AgentRuntime spec.podOverrides |
| session-api | Workspace spec.services[].session.podOverrides |
| memory-api | Workspace spec.services[].memory.podOverrides |
| Eval-worker | Workspace spec.services[].evalWorker.podOverrides |
| Eval-worker for the agent’s service group | AgentRuntime spec.evals.podOverrides (the Workspace field takes precedence) |
| Arena workers | ArenaJob spec.workers.podOverrides |
Mount CSI-synced secrets into session-api
Section titled “Mount CSI-synced secrets into session-api”-
Add
podOverridesto the service group’ssessionblock. This example uses Azure workload identity and the Secrets Store CSI driver. The values are illustrative.spec:services:- name: defaultmode: managedsession:database:secretRef:name: session-dbpodOverrides:serviceAccountName: workload-identity-saannotations:azure.workload.identity/use: "true"extraVolumes:- name: kv-secretscsi:driver: secrets-store.csi.k8s.ioreadOnly: truevolumeAttributes:secretProviderClass: session-db-spcextraVolumeMounts:- name: kv-secretsmountPath: /mnt/secrets-storereadOnly: trueextraEnvFrom:- secretRef:name: session-db # created by the SecretProviderClass secretObjects -
Apply the Workspace.
Terminal window kubectl apply -f workspace.yaml
Run an agent on GPU nodes with its own identity
Section titled “Run an agent on GPU nodes with its own identity”-
Add
podOverridesto the AgentRuntime. This example uses AWS IRSA, an NVIDIA node label and a private registry. The values are illustrative.apiVersion: omnia.altairalabs.ai/v1alpha1kind: AgentRuntimemetadata:name: vision-agentspec:promptPackRef:name: visionfacades:- type: websocketpodOverrides:serviceAccountName: vision-irsanodeSelector:nvidia.com/gpu.product: A100tolerations:- key: nvidia.com/gpuoperator: Existseffect: NoScheduleimagePullSecrets:- name: ecr-pull -
Apply it.
Terminal window kubectl apply -f vision-agent.yaml
To set affinity, use spec.runtime.affinity. podOverrides has no affinity field.
Give Arena workers shared credentials
Section titled “Give Arena workers shared credentials”-
Add
podOverridesunderspec.workers:apiVersion: omnia.altairalabs.ai/v1alpha1kind: ArenaJobmetadata:name: eval-run-1spec:sourceRef:name: my-arena-sourceworkers:replicas: 5podOverrides:extraVolumes:- name: provider-keyscsi:driver: secrets-store.csi.k8s.ioreadOnly: truevolumeAttributes:secretProviderClass: arena-providersextraEnvFrom:- secretRef:name: arena-provider-keys -
Apply the ArenaJob.
Verify
Section titled “Verify”Find the Deployment for the workload, then read the ServiceAccount and volumes the operator rendered.
kubectl get deployments -n <namespace>kubectl get deployment <name> -n <namespace> \ -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}{.spec.template.spec.volumes[*].name}{"\n"}'The output lists your serviceAccountName and the volume names you added. For an ArenaJob, inspect a worker pod with kubectl get pod <pod> -o yaml.
Troubleshooting
Section titled “Troubleshooting”A label you set is missing
Section titled “A label you set is missing”Operator-set labels win on key collision because Service selectors depend on them. Use a different label key.
An environment variable is missing from the agent pod
Section titled “An environment variable is missing from the agent pod”The operator drops reserved keys such as OMNIA_PACK_MODE from extraEnv and sets the ReservedEnvClean condition to False.
kubectl describe agentruntime <name>The agent runs as the wrong ServiceAccount
Section titled “The agent runs as the wrong ServiceAccount”An AgentRuntime’s ServiceAccount is podOverrides.serviceAccountName, else the Workspace’s spec.runtime.serviceAccountName, else <agentruntime>-facade.